Jens Gustedt's Blog

About a year ago I posted about defer and that it would be available for everyone using gcc and/or clang soon. So it is probably time for an update.

Two things have happened in the mean time:

• A technical specification (TS 25755) edited by JeanHeyd Meneide is now complete and moves through ISO’s complicated publication procedures.
• Both gcc and clang communities have worked on integrating this feature into their C implementations.

I have not yet got my hands on the gcc implementation (but this is also less urgent, see below), but I have been able to use clang’s which is available starting with clang-22.

I think that with this in mind everybody developing in C could and should now seriously consider switching to defer for their cleanup handling:

• no more resource leakage or blocked mutexes on rarely used code paths,
• no more spaghetti code just to cover all possibilities for preliminary exits from functions.

I am not sure if the compiler people are also planning back ports of these features, but with some simple work around and slightly reduced grammar for the defer feature this works for me from gcc-9 onward and for clang-22 onward:

#if __has_include(<stddefer.h>)
# include <stddefer.h>
# if defined(__clang__)
#  if __is_identifier(_Defer)
#   error "clang may need the option -fdefer-ts for the _Defer feature"
#  endif
# endif
#elif __GNUC__ > 8
# define defer _Defer
# define _Defer      _Defer_A(__COUNTER__)
# define _Defer_A(N) _Defer_B(N)
# define _Defer_B(N) _Defer_C(_Defer_func_ ## N, _Defer_var_ ## N)
# define _Defer_C(F, V)                                                 \
  auto void F(int*);                                                    \
  __attribute__((__cleanup__(F), __deprecated__, __unused__))           \
     int V;                                                             \
  __attribute__((__always_inline__, __deprecated__, __unused__))        \
    inline auto void F(__attribute__((__unused__)) int*V)
#else
# error "The _Defer feature seems not available"
#endif

So this is already a large panel of compilers. Obviously it depends on your admissible compile platforms whether or not these are sufficient for you. In any case, with these you may compile for a very wide set of installs since defer does not need any specific software infrastructure or library once the code is compiled.

As already discussed many times, the gcc fallback uses the so-called “nested function” feature which is always subject of intense debate and even flame wars. Don’t worry, the implementation as presented here, even when compiled with no optimization at all, does not produce any hidden function in the executable, and in particular there is no “trampoline” or whatever that would put your execution at risk of a stack exploit.

You may also notice that there is no fallback for older clang version. This is because their so-called “blocks” extension cannot easily be used as a drop-in to replace nested function: their semantics to access variables from the surrounding scope are different and not compatible with the defer feature as defined by TS 25755.

So for example if you are scared of using big VLA on the stack, you may use the above code in headers and something like

double* BigArray
  = malloc(sizeof(double[aLot]));
if (!BigArray {
  exit(EXIT_FALURE);
}
defer { 
  free(BigArray); 
}

to have an implementation of a big array with a failure mode for the allocation.

Or if you want to be sure that all your mutexes are unlocked when you leave a critical section, use and idiom as here

{
  if (mtx_lock(&mtx) != thrd_success) {
    exit(EXIT_FAILURE);
  }
  defer {
    mtx_unlock(&mtx);
  }

  ... do something complicated ...

  if (rareCondition) {
    return 42;
  }

  ... do something even more complicated ...
}

Just notice, that you’d always have to use the defer feature with curly braces to ensure that the gcc fallback works smoothly.

Since the last meeting of the C committee I am struggling with the idea of aliases as they are proposed in a long series of papers by JeanHeyd.

Transparent Aliases

I am struggling with this for several reasons, but most importantly is that I have the impression that it is a heavy gun pointed on a problem that merely seems to be about tooling, or at least is presented as such. And then, there is this rejection of macro solutions that drives me nuts. That looks just like bashing to me, evacuation a possible solution without giving if the necessary thoughts.

_Alias(memcpy, my_meme);

(the proposed syntax is a bit different, but let’s not go into that for now)

This declares an identifier memcpy for the rest of the scope that is as good as using my_meme everywhere. This then could be used for configuring a source for example with a #if cascade that chooses the replacement according to some word size, target architecture capacity or so. Once compiled, the resulting binary would then always use the same function my_meme, even if the system’s memcpy is linked dynamically.

If we see this as a simple replacement of identifiers (which it isn’t) we could be tempted to just declare a macro memcpy that resolves to the concrete choice I want to make at that point in the source. And that is indeed the way that is foreseen by the C library: any C library function may be implemented as a macro that points to the stuff that we need.

Problems with the usual macro approaches

There are several problems with this approach.

First, we’d have to ensure that macro quacks like a function descriptor, that is

• usages with parenthesis and arguments should result in a call,
• usages without parenthesis should convert to a function pointer,
• applying the & operator should convert to a function pointer,
• applying the * operator should give back the function descriptor.

So, first the only choice that leaves for a macro is to have it “object-like”, that is without parenthesis in the definition.

Then, there is the problem of locality of definition. We’d want to choose at the point of definition, to which other symbol my macro resolves. A simple macro replacement takes whatever symbol my_meme is defined at the point of macro invocation. That could be something completely different than what the coder of the macro had in mind.

Another problem is the one of scope. If I define my replacement just within a code block, I wouldn’t want the macro to pollute the code that follows that block and randomly replace things there.

Add to that the usual problem of macros, that they also replace names in other name spaces, for example structure members, or where the symbol would locally be used otherwise, for example a function parameter named memcpy. But that is clearly something that we can’t change easily. It is inherent to the macro approach and C programmers should know how to deal with this.

A serious approach to provide aliases with macros

A spelled out solution to do aliases with macros could look as follows

auto const __memcpy_alias_7854 = &(my_meme);
#define memcpy /*space*/ (*__memcpy_alias_7854)
...
#undef memcpy // only if used inside a {} block

So first of all, we provide ourselves with an immutable variable that holds the address of the symbol which we are after. Then we ensure that the identifier memcpy will always resolve back to the original function descriptor. Now

• memcpy(something) will call my_meme.
• memcpy without parenthesis is (*__memcpy_alias_7854) a function descriptor that decays to a function pointer, just as a direct use of my_meme would.
• &memcpy is &(*__memcpy_alias_7854) where &* cancel out and the result is a function pointer, just as a direct use of my_meme would.
• *memcpy is *(*__memcpy_alias_7854) which first decays (*__memcpy_alias_7854) to the function pointer and then applies * to result in a function descriptor, again, just as a direct use of *my_meme would.

So in all, this approach solves most of the inherent problems of doing this with a macro.

Some remarks and observations:

• The ergonomics of that approach could be better. You’d have to implement two or three consistent lines, watch about the space between the name and the opening parenthesis, and in addition you need to invent a random, unique name for the pointer. Not beautiful.
• This uses C23’s auto feature. If you don’t have that (yet) you’d have to know your type or use an extension such as gcc’s __typeof__ for the declaration.
• You’d probably like the auxiliary variable to be as “optimizable” as possible. So in block scope register would be in order. If you are in file scope, you wouldn’t want the auxiliary name pollute your linker space, so you’d probably have that as static.
• Any decent modern compiler with some optimization switched on should be able to make the use of the function pointer transparent and optimize it away completely.

So, if you are willing to accept the bad ergonomics, this is a solution for the problem.

An encapsulation with eĿlipsis

Enters eĿlipsis, obviously I can’t let such a solution out, without programming some improvement into my preprocessor. In the next release, there is a new header <ellipsis-reference.h> that implements a macro _Alias that follows that idea: it introduces a local alias for an addressable lvalue or function descriptor and defines a macro that is bound to the current C scope. So the behavior is as-if when this is used inside some pair of {}, an #undef of the macro is inserted at the closing brace.

The second argument (and possible further arguments all together) have to expand to an lvalue or function descriptor. Consider the following example with a user function my_strlen.

char* fun(char x[]) {
   _Alias(strlen, my_strlen);
   …
   size_t len = strlen(x);
   …
}

Here all invocations of the strlen object-like macro inside the function are replaced by (*_Alias_strlen_34) as follows. (The 34 is just a number to ensure that the identifier is unique.)

char* fun(char x[]) {
   [[maybe_unused]] register auto const _Alias_strlen_34 = &(__my_strlen);
   …
   size_t len = (*_Alias_strlen_34)(x);
   …
}

Note that in addition to the above, this is automatically annotated with [[maybe_unused]], so unused aliases in headers wouldn’t trigger warnings, and with register such that the address of _Alias_strlen_34 can never be taken. Unfortunately the latter does not work in file scope, so there static is used instead.

Possible use cases for _Alias

The use of _Alias goes far beyond the initial motivation for finding a tool to freeze architectural choices consistently to a given situation. Even with JeanHeyd’s feature we could give the name of a variable for the RHS and thus have an alias refer to an object. With eĿlipsis we can even do more, something like

char* fan(double (*Ap)[42]) {
   _Alias(A, *Ap);
   …
   for (size_t j = 0; j < 23; ++j) {
       for (size_t i = 0; i < _Countof(A); i++) {
           A[i] += i;
       }
       ... do something else ...
   }
}

This would provide us with the possibility to see a function parameter as some sort of reference and in particular handle parameter arrays in a convenient way.

I am not a fan of that approach as shown above, because it is, as the name indicates, an alias and not a reference. An alias provides a new name for an existing feature but it does not ensure that the feature is not accessed through other channels. Much of C and its optimization capabilities come from the fact that it avoids such aliases and that the compiler can safely assume that they have the only view on an object. In the example above the line

           A[i] += i;

might allow for very different optimizations, when there is a guarantee that the view through A is the only one, than when it is not.

Lvalue references

So a reference (as provided by many programming languages other than C) is more than an alias, it guarantees

• that there is an object, and
• that, in the given scope, the access to this object is unique.

These assumptions allow the compiler to do a lot of optimizations that otherwise would not be possible.

As you probably know, C does not have that concept, and this is a conscious choice. Many C programmers become very passionate about this, the mere possibility of aliasing or hiding pointers from view can put us into crisis mode.

A macro version of lvalue references for function parameters

The eĿlipsis header <ellipsis-reference.h> has another macro for parameter declarations that avoids some of these problems namely _LVparam. Other than _Alias it is just applied to the name of a parameter. Taken the fan example from above that would be

char* fan(double _LVparam(A)[42]) {
   …
   for (size_t j = 0; j < 23; ++j) {
       for (size_t i = 0; i < _Countof(A); i++) {
           A[i] += i;
       }
       ... do something else ...
   }
}

that is expanded to something like

char* fan(double (_LVparam_A_16[restrict const static 1])[42]) {
   …
   for (size_t j = 0; j < 23; ++j) {
       for (size_t i = 0; i < _Countof((*_LVparam_A_16)); i++) {
           (*_LVparam_A_16)[i] += i;
       }
       ... do something else ...
   }
}

So almost the same as before, but not quite. The difference is that the _LVparam_A_16 pointer parameter is given as an array with additional annotations. (Remember that C has the crude rule that the innermost array bound of a parameter is rewritten to a pointer.)

• restrict says that the pointer is the only possible way to access the pointed-to object inside the function.
• static 1 says that the pointed-to object has at least one element. So in particular, the pointer is never null.

So with these additional annotations, A now is really a reference to a vector object. But be careful, whereas the compiler may now make a uniqueness assumption inside the function, there is no guarantee in C that this property is checked on the caller side. Compilers are getting better with this, but there are situations where they just can’t tell. Use your favorite static analyzer to obtain the most possible guarantees.

The _LVparam feature has the necessary properties to be applicable to real code, namely the macro definition of the parameter name extends from the definition to

• the closing brace of the function body if the function declaration is also a definition;
• to the terminating semicolon if the declaration is a forward declaration of the function.

A macro version of lvalue pseudo-references for non-parameter objects

Unfortunately, the static 1 notation as used above only works for array parameters (that are then rewritten to pointers). For references that we would like to define inside functions such a feature is not available. Nevertheless, we can use a similar feature for local lvalue references, _LVlocal,

double _LVlocal(A)[len] = calloc(sizeof A, 1);

This expands to something similar to

double (*restrict const _LVlocal_A_12)[len]
   = calloc(sizeof(*_LVlocal_A_12), 1);
if (!LVlocal_A_12) exit(EXIT_FAILURE);

Here we directly declare a pointer and again, restrict makes the promise that this pointer is unique. Unfortunately, there is currently no direct syntax to also ensure that the pointer is non-null. So the tool inserts the check automatically at the next semicolon in the source.

If we then even add an automatic destruction that triggers when the scope is left

double _LVlocal(A)[len] = calloc(sizeof A, 1);
defer { free(&A); };

we have implemented a stack-safe version of a variable length array, VLA that checks automatically if the allocation succeeded. This solution needs that your compiler implements the new defer feature or that you are using the one from eĿlipsis.

Another possible solution for the problem that the allocation of VLAs may not succeed would be not to provide the implicit test as it is implemented here, but to leave it to the user code to test for it simply by testing if &A is null.

Conclusion

C already has the semantics to cover features such as JeanHeyd’s proposed _Alias and more generally to implement references of objects or VLA that include runtime checks instead of smashing your stack. I personally am not yet completely convinced that I want any of this in my code, but we have it now in eĿlipsis. These features are not completely equivalent to proper language features (macro declarations in eĿlipsis still can’t shaddow one another, for example) but it comes quite close and should be good for at least 90% of the use cases.

So please give it a try if you like these kind of things.

At last, the new edition is now also available in print and e-pub versions, the contents is the same as described in the previous article.

Find all the details at

https://gustedt.gitlabpages.inria.fr/modern-c/

[The wordpress markdown inclusion does a very bad job, it seems, there have been some idiotic formatting errors. I hope that these are fixed, now.]

A years-long effort led by Kayvan Memarian and Peter Sewell from Cambridge University, UK, Martin Uecker from Graz University of Technology, Austria, and myself (from ICube/Inria, France) has guided the C community to accept a common vision of so-called pointer provenance, defining how to trace the origins of pointer values through program execution. Our provenance-aware memory object model for C provides a precise mathematical specification, in place of the ambiguity of these aspects of the current C standard. It has also stimulated and informed discussion of provenance in the broader C, C++, Rust, and compiler communities.

This work has finally resulted in the publication of an international standard, Technical Specification ISO/IEC TS 6010 (edited by Henry Kleynhans, Bloomberg, UK). With the goal of making modern information systems safer and more secure, this official technical specification provides direction to all stakeholders in the industry such that they can converge their platforms and tools.

In this article, I will try to explain what this is all about, namely on how a provenance model for pointers interferes with alias analysis of modern compilers. For those that are not fluent with the terminology or the concept we have a short intro what pointer aliasing is all about, a review of existing tools to help the compiler and inherent difficulties and then the proposed model itself. At the end there is a brief takeaway that explains how to generally avoid complications and loss of optimization opportunities that could result from mis-guided aliasing analysis.

Pointer aliasing and program optimization

We say that two pointer values p and q during the execution of a program alias if they point to the same object in memory.[1] To see that the question if two pointers alias has an influence on the optimization of code, let’s consider the following simple example of an iterative function.[2]

---

1. Or more precisely if the objects pointed-to have at least one byte in common. ︎

2. Please, please, don’t take the function as given here as a basis for your code. It is only chosen for illustration of the points that I want to make here about aliasing. ︎

It implements an approximation algorithm for the reciprocal r := 1/a of the value a := *aₚ which doesn’t use division.

// Reciprocal approximation
//
// This interface is horrific, don't use it.
// This just serves as an artificial example
// for this article.

constexpr double ε  = 0x1P-24;
constexpr double Π⁻ = 1.0 - ε;
constexpr double Π⁺ = 1.0 + ε;

void recip(double* aₚ, double* řₚ) {
    for (;;) {
        register double Π = (*aₚ)*(*řₚ);
        if ((Π⁻ < Π) && (Π < Π⁺)) {
            break;
        } else {
            (*řₚ) *= (2.0 - Π);
        }
    }
}

The function receives a pointer to a second value ř := *řₚ with a rough approximation for r. It then iteratively approaches r within a chosen precision ε: the current values a and ř are multiplied into a value Π and if that value is sufficiently close to 1.0 the iteration stops. If it is not, ř is corrected and the loop continues.

What is interesting for our context of aliasing is that this function has two pointer arguments that both point to a value of the same type double. One of these pointer targets *řₚ is loaded from memory, modified and stored at each iteration. In total, the non-optimized function as specified above in each iteration has

• 3 load and 1 store operations,
• 2 comparisons,
• 1 logical operation, and
• 3 arithmetic operations.

So loads and stores from memory make up 4 of about 10 operations in total.

But wait, can’t this be done better? Yes, obviously, a much better version of this could look as follows.

void recip⁺(double* aₚ, double* řₚ) {
    register double a = *aₚ;
    register double ř = *řₚ;
    for (;;) {
        register double Π = a*ř;
        if ((Π⁻ < Π) && (Π < Π⁺)) {
            break;
        } else {
            ř *= (2.0 - Π);
        }
    }
    *řₚ = ř;
}

That is, we load a and ř once, at the beginning, and then only update *řₚ when we have finished our computation. The hope here is that these new local variables use “hardware registers” that can be used directly by the processor, without going through loads and stores from and to the program memory.

So roughly the optimized function saves us 40% of the operations. This means that the optimized function is in general much faster and achieves its goal by wasting less energy.

Unfortunately no C compiler can do this optimization automatically:

The functions recip and recip⁺ and not equivalent.

We didn’t see this, yet, because we failed to discuss an important feature of the original program; the pointers aₚ and řₚ may either point

• to different objects, or
• to the same object.

Indeed, our optimized function recip⁺ only covers the first of these possibilities and not the second: the second case provides a completely different algorithm for which I wouldn’t know any use or properties. But since the compiler can’t know which of these cases we have in mind (maybe both?) it cannot do an optimization that excludes the second.[1]

In general, loads and stores from memory are expensive operations, so a lot of effort in modern compiler frameworks goes into so-called alias analysis, that is in an detailed analysis of which pointers may alias each other (or not). Such alias analysis then can be used to mechanically prove whether a specific optimization is feasible or not.

---

1. Any modern compiler will still do some optimization with recip, but would usually not arrive at the same level of improvement. ︎

Good alias analysis of pointer targets is crucial for optimization in modern compilers.

In the case of recip we see that our specification is not precise enough to know whether or not the second case (where the pointers alias), can be excluded. We also see that misguided assumptions about pointer aliasing can result in implementing a completely different algorithm. Or, in other words, they may result in a severe bug.

Mislead aliasing assumptions result in bugs.

In one of our papers, we provide a long list of examples where pointer aliasing can lead to different interpretations by users and compilers, see also this paper here.

Note also that a previous attempt by the C committee to introduce stricter aliasing rules had turned into years of flame wars between alienated parts of the user community, compiler builders and committee members. A communication disaster that was very harmful for the industry in general and to the trust of the community in the C committee in particular.

Helping the compiler

Modern compilers have several mechanisms that help to deduce more information about the intended use cases for pointers and from there to provide correct optimizations. These mechanisms are

• Type-based alias analysis.
• Flow-based alias analysis.

For type-based alias analysis, C takes a relatively simple approach, at least on the surface level.

Two pointers that point to objects of different non-character type are supposed not to alias.

So, for our function recip above, it would have been sufficient to have arguments pointing to differently typed objects. Then, an optimization similar to recip⁺ would have been valid.

But …

… notice the “supposed” in the phrase above. In C, it is entirely the programmers responsibility to ensure that two differently typed objects are in fact different. If you are casting spells to convert pointers from one target type to another, you have to prove for yourself that you may pass such pointers as arguments to functions without causing problems, there. You are completely on your own.

Flow-based analysis in C is supported by several tools

• The register storage class.
• The restrict pointer qualifier.
• The volatile qualifier on the target type.

The first ensures that the address of a so-declared object is never taken. And in C, when there are no pointers, there is no aliasing.[1] Although this is easy enough to use and provides feedback if one tries to obtain the address of such an object, it seems that this is much less used than it should.

---

1. This might be completely different in other programming languages. E.g in C++ there is a concept of “references”, that when passed around into functions, result in similar questions on aliasing as we see here for pointers in C. ︎

The second is much more subtle. If a pointer parameter of a function is declared with restrict, every user of the function has to ensure that the pointer value is the unique view on the object from within for each call to the function. The definition in the standard of that feature is obscure and, again, it has the same drawback as type-based alias analysis in C: you are completely on your own.

The third is a feature that imposes that each access to a volatile qualified object reloads the object from memory, regardless what the compiler might know about its previous value. Using this feature excessively is like stepping on the break: basically all optimization opportunities concerning that object are lost.

As extensions, some compilers also have command line flags that help the user to steer through. For example gcc has the options -fstrict-aliasing and -fno-strict-aliasing to switch aliasing analysis as defined by the C standard on or off.

And then, there is provenance …

In the current C standard provenance is only a hidden concept, the word “provenance” appears in it exactly 0 times. Nevertheless, it marks an important assumption for existing C, namely that somehow compilers are allowed to assume that two pointers don’t alias when they are known to come from different “sources” in the program.

For programmers, the current state is a disaster. In many cases there is no way to know which assumptions a compiler makes:

• There is no way (but restrict and types, see above) to claim that two pointers never alias.
• It is difficult (but for volatile) to specify that they might alias such as to force optimization to be more restricted, either. This even in situations where, for example, the base types are in fact different, but where the user knows that the underlying object might still be the same.

The concept of provenance created many difficulties because it was left undefined exactly what it meant, so each compiler implementer had their own ideas how to assert that two pointers do (or don’t) alias, and how to use that in optimization.

When digging deeper we observed several problem spots, and it took us in fact a long time to understand the different assumptions.

Object granularity

The first question that comes up is to figure out at which granularity we can do or want to do aliasing analysis. At the level of bytes, words, basic type objects, structure objects, memory allocations, or, the whole address space?

The main problem here is that in C we can move from one pointer address to the next via arithmetic. The easiest examples are arrays. If we have access to an element in the middle of an array, we can move back and forth by simply adding or subtracting an integer value to the corresponding pointer value. How would we (the programmers) or the compiler know if two pointers point into the some array but at different elements? What happens if we start to step backwards from such pointers?

This picture becomes even more confusing, if we allow pointer arithmetic on the byte level. In C, the offsetof macro allows us to access arbitrary members of a structure by using such pointer arithmetic.

Pointer equality and object life time

When you are programming with pointers in C, you are hopefully aware of the “dangling pointer” problem. This refers to the fact that for example a pointer to a local variable becomes invalid once you leave the scope (e.g. function) where it is declared.

For aliasing analysis, this becomes even more complicated: not only does a pointer become invalid, the address of the dead object may be reused in a different context. So two pointers may even be equal, one pointing to a dead object, the other to a new one.

There is another situation where two pointer values are the same, but where they talk about two different objects. This occurs when two arrays, A and B, say, happen to live in adjacent memory locations. If A has n elements, the pointer value &A[0]+n (the end of A) might be the same as &B[0] (the start of B). So then we have two pointer values that are the same but that have quite a different meaning for the programmer.

Information flow for addresses

Another concept influences aliasing analysis a lot, namely the possibility that a pointer value escapes from a limited scope of the program (e.g the address of a local variable) and becomes known in other parts of the program.

For example a variable that is declared static in a block is usually only visible there. In general the compiler masters very well which pointers may alias with a pointer value that corresponds to the address of that static variable. If the address of that variable “escapes” from the local scope, for example by passing it as an argument to a function, the address could come back into the same scope via unknown paths that are difficult to control.

There are surprisingly many ways that pointer values as a whole or parts of them may escape from one context and reappear in another. Among them are

• manipulations of the byte-representations of pointers for example
  • copying by memcpy,
  • copying by bytes,
  • IO using %p in printf or
  • fwrite
• manipulation of integers that are the result of pointer-to-integer conversions.

Perhaps surprisingly the latter is an important point that had to be handled carefully in our proposal. The reason is that pointer bit-manipulations are used in contexts where the available memory relatively constrained, such as systems programming for example. Bit-manipulation tricks are then used to save on the size of data structures. This has not only the advantage that storage space for the data can be reduced, but also that a reduced size also improves the performance for data that are used intensively due to improved use of processor caches.

A famous example is the XOR trick for doubly linked lists, where a data structure stores the XOR of the bit patterns of two pointer values. So for that particular use we have an integer (here of type uintptr_t) that contains information that comes from two different pointers, that then is used to sometimes reconstruct one or the other of these pointer values.

typedef struct elem elem;
struct elem {
    uintptr_t both;
    double data;
};

void elem_store(elem* e, elem* prev, elem* next, double val) {
    (*e) = (elem){
        both = (uintptr_t)prev ^ (uintptr_t)next,
        data = val,
    };
}

Such a data structure can then be used for elements of a list that can be traversed consistently forward and backward, but the memory footprint is only that of a simply-linked list:

elem* elem_next(elem const* e, elem* prev) {
    return (elem*)((uintptr_t)prev ^ e->both);
}

elem* elem_prev(elem const* e, elem* next) {
    return (elem*)((uintptr_t)next ^ e->both);
}

As you can see, besides their names these two functions are completely identical, and I can’t imagine any compiler being able to track the origin of a pointer that one of these two functions returns.

Tracking provenance through integers?

The fact that in C pointer values are closely related to integers creates a lot of confusion. In the case of aliasing analysis, the lack of separation between those terms has it that we have to integrate a model of information flow of pointer values (or just some bits or bytes of them) through integers and back to pointer values.

When Peter and Kayvan started their investigations, they had to consider different possibilities, one of them being to track information about pointers through such a chain of conversions. It turned out, that such a model would be possible (they provided a sound specification for it) but that it would come at an important cost. For code as for the XOR trick used above a pointer value (the result of a call to elem_next for example) would have two origins (prev and next in a call to elem_store) and not only one. Since such different origins could then accumulate if we do more operations, basically an integer and a pointer derived from it, could have an arbitrary number of origins.

Such a model with multiple origins of pointers seemed complicated and impractical. Complicated for users, because they would have to be aware that information about pointers could be used in surprising ways by a sophisticated compiler. Impractical for compilers because keeping track of all possible in-flow of information would result in a combinatorial explosion of the state of the abstract machine.

Presented with such a complication, one possibility would have been to just “forbid” using pointers in that way. We could have stated something along the lines of “if a pointer has several origins, the behavior is undefined” and thus leaving everything (the “undefined” part) to compiler implementations. But because these situation appear in real life code, this would have left these important parts non-portable between different compilers and architectures.

So the overall conclusion was not to ban such usage of pointers through integers, but to formalize it and label such pointers as exposed, that is that no user has to fear that compilers will present them with optimization that uses knowledge of such information flow, and no compiler has the pressure to optimize such code, either.

The provenance model

The provenance model that we came up with, and which is at the base of TS 6010, tries to take all of these aspects into account with the goal to provide something that at the same time can easily be referred to by users and compiler implementers. It provides some compromise between the expectations of the two communities in the sense that it does not leave all the liberty they might dream of to compiler providers for optimization, and it still has some sort of complexity and difficulty for users.

In the following we present the most important parts of that model.

Storage instance

As said above, we have to agree upon the granularity of memory accesses for which aliasing of pointers will be considered. When we combed through the existing standard (C11 an C17 at the time) we quickly noticed that there was not even agreement within that standard. When it talks about what is found at the other end of a pointer, it talks about “object”,[1] “space”, “memory” or “storage” and even some combinations of these.

---

1. Confusingly, there is another notion of object related to types that is used throughout the C standard, but which is not the same as an “allocated object” as we are talking, here. ︎

It seemed important to us to emphasize that pointers and addresses are already an abstraction that does not necessarily denote a physical device: most modern platforms nowadays form so-called virtual address spaces. Such “virtual addresses” then are in general transformed by low-level tools to “physical addresses” that represent concrete memory hardware. To make that distinction clearer we decided to use the term “storage” in most places where one of the terms noted above appear.

Another important observation to have is that we even have to talk about things that do not have an address. For example if we declare a variable width the register storage class, we cannot receive a pointer to that object and the whole point is that it is not necessary for the compiler to realize this variable in the main memory.

Then, the aspect of temporality also comes into play. A chunk of memory that is obtained for example by a call to malloc can be returned to the system by calling free and then might again be served by another call to malloc. It is important that the two entities to which the pointer refers are seen as completely different and that the fact that they reside in the same memory location is a simple coincidence.

For the granularity, we decided to go on the level of maximal region in which “legally” a C program could operate. Since inside any allocation or declared object all bytes are reachable by character-pointer arithmetic, we decided to take this as the level of granularity. Therefore

A storage instance is the maximal region of storage that is provided by

• an allocation (malloc and similar),
• a variable definition,
• a compound literal, or
• an object with temporary lifetime.

Note here that the second point talks about the definition of a variable, not its declaration. For local variables these two coincide, but for file scope variables (outside any function) there can be declarations (with extern) that are not definitions. The definition of a variable is always unique and specifies where it is located, namely in our terminology here, where the storage instance that represents it comes to be.

If a storage instance is addressable, the conversion of a pointer to its start to the type unsigned char* points into an array, called its byte array, where each element is one byte of the storage instance.

By that definition, conversions of pointers to character types and to void* are defined and it is uniquely prescribed how arithmetic on a byte level works.

A storage instance has a lifetime that expands

• from the allocation (typically malloc) to the deallocation (typically free)
• from the definition of the variable to the point where the block of the definition left (for a VLA)
• from the point of entering the block of the definition until it is left (for other variables with automatic storage duration)
• from the point of entering the innermost block that contains the expression until that scope is left (for compound literals with automatic storage duration)
• from the start of the program execution until its end (for static objects)
• from the start of the thread execution until its end (for thread_local objects)
• during the evaluation of the full expression that contains it (for objects of temporary lifetime).

So in addition to the “where” above, this definition describes when the storage instance that represents an object comes to be and ceases to exist.

If you are not familiar with all the concepts in that item list, just ignore these. The importance here really is to make it clear for the features that you use in your program and know about, that in general their lifetime is limited and that any such allocation or definition gives rise to one single storage instance per context in which the construct is met.

For example in the following code we see three storage instances in action

{
    size_t n  = 32;
    double (*A)[n][n] = malloc(sizeof(*A));
    ...
    free(A);
    A = nullptr;
    ...
}

The two obvious ones are

• The one for n, a variable of integer type, and for which the lifetime starts when we enter the block at the { and that ends when we leave it at }.
• The storage instance that is allocated by the call to malloc and that is deallocated by the call to free.

But then there is also a storage instance for the pointer A itself that, similar to n, lives during the execution of the block. In particular, after we freed the contents of A we may still access it to set it to a null pointer value.

For a case where the notion of storage instance is perhaps a bit less intuitive we note that calls to realloc are a bit peculiar with respect to that definition. In a call

void* p = realloc(q, 77);

we first have the storage instance to which q points. Then, if the call is successful, that old storage instance is released and a pointer to a new storage instance is stored in p. Even if these two pointers are identical (possibly the storage instances start a the same address) they are nevertheless considered as two different entities.

With the term storage instance immediately comes the notion of provenance.

The provenance of a valid pointer value is the storage instance into which (or one beyond which) the pointer value points.

With the exception of one particular border case (see below) the provenance of a valid pointer value is unique.

Address space

To be useful in an aliasing model, the concept of an address space is not provided with enough precision in the current C standard. We need to talk consistently about addresses, how pointers convert, compare or relate.

The model we came up with, has the following properties:

• To each object pointer value corresponds an abstract address that is a positive integer value.
• Bytes within an addressable storage instance have abstract addresses that increase from start to end.
• If the distinct storage instances A and B are alive at a common point in time, the abstract addresses of all bytes of A are either all smaller or all greater than the abstract addresses of all bytes of B.
• Two pointer values are equal if they correspond to the same abstract address.
• One pointer value is smaller than another pointer value, if both point into the same storage instance and if the address of the first is smaller than the one of the second.
• If the platform is able to represent all addresses in some integer type, the type uintptr_t is provided and a conversion from a pointer to that type provides the abstract address.
• Conversion from pointers to any integer type are consistent with that mapping to abstract addresses.

This model falls short from defining a “flat” address space:

• Arithmetic on pointers and arithmetic on abstract addresses need not to be consistent.
• Even within the same storage instance, the increase from one byte to next may not be one, and may not even be uniform.
• The type uintptr_t may not exist.
• Conversion to integer types that are too narrow has undefined behavior.

The reasons for only having such a lax definition are simple, for each of the weird properties in the list there are examples that make it necessary. In particular, there are platforms with segmented memories that have “bumps” in the address space, and platforms that pack additional bits into pointer values that are not related to the corresponding abstract address.

Exposure and synthesis of pointer values

Another observation is crucial for our model: most aliasing analysis isn’t perfect. That is, compilers as well as programmers have limits of which tracks of the pointer information they can follow. For example the XOR trick shows that a pointer value can have several origins. In all we have to foresee a mechanism that describes the boundaries of the assumption that a compiler may make on one hand, and the guarantees that a programmer has to give on the other.

The mechanism with which we came up has two sides

• A pointer value is exposed if information about its abstract address or its in-memory representation leaks to the outside or to distant parts of the program.
• A pointer value is synthesized if it is assembled from outside information, from byte information or from integer values.

The goal is to describe that mechanism in a way such that (in principle) some auxiliary information could be added to each pointer value that would either allow

• compilers and users to establish aliasing properties of a set of pointer values, or
• easily come to a consensus for situations for which such analysis is abandoned an may not be assumed.

Exposure

Let’s now have a look at a possible normative text as it should be integrated into the C standard at some point

A storage instance becomes exposed when a pointer value p of effective type T* with this provenance is used in the following contexts:

• Any byte of the object representation of p is used in an expression.
• The byte array pointed-to by the first argument of a call to the fwrite library function intersects with an object representation of p.
• p is converted to an integer.
• p is used as an argument to a %p conversion specifier of the printf family of library functions.

The idea of the first bullet point is that if we read bytes of the object representation of a pointer, cascaded if/else control flow could be used to reconstruct pointers and thus jeopardize any aliasing analysis. But what we also didn’t wanted, is that “normal” operations that programmers do on pointer representations as a whole would have similar effect as access.

Therefore notice that memcpy or similar functions do not appear in the list above; as long as we use it to copy pointer representations as a whole, provenance can simply be transferred. For example, using memcpy on structure objects that have pointer members is fine: such an operation copies the whole pointer without exposing individual bytes. So we simply assume that the provenance information is transferred at the same time to the target structure.

We also don’t want that small changes in the way that we look at a pointer representation has an influence on aliasing analysis. Therefore we add the following paragraph

Nevertheless, if the object representation of p is read through an lvalue of a pointer type S* that has the same representation and alignment requirements as T*, that lvalue has the same provenance as p and the provenance does not thereby become exposed.

Here the term “same representation and alignment” covers for example the possibility to look at

• a pointer that has different qualifiers than the original,
• where one type would be a signed type and the other unsigned, or
• one would be a structure and the other would be another structure that sits at the beginning of the first.

Also exposure is meant to be a one-way street, once exposed we will never know where the information about the pointer could creep into our program execution.

Exposure of a storage instance is irreversible and constitutes a side effect in the abstract state machine.

Synthesis

The inverse operation that uses other information to produce a pointer looks as follows:

A pointer value p is synthesized if it is constructed by one of the following:

• Any byte of the object representation of p is changed
  • by an explicit byte operation,
  • by type punning with a non-pointer object or with a pointer object that only partially overlaps,
  • or by a call to memcpy or similar function that does not write the entire pointer representation or where the source object does not have an effective pointer type.
• The object representation of p intersects with a byte array pointed-to by the first argument of a call to the fread library function.
• p is converted from an integer value.
• p is used as an argument to a %p conversion specifier of the scanf family of library functions.

Additionally, we always require that the storage instance that is synthesized has been exposed before.

Ambiguities

With all of that the situation about adjacent storage instances still remains. That is, a situation where two arrays A and B are adjacent in memory. Let’s suppose the two arrays have two elements, each, and that the base type has four bytes:

A[0] A[1] B[0] B[1] a₀ a₁ a₂ a₃ a₄ a₅ a₆ a₇ b₀ b₁ b₂ b₃ b₄ b₅ b₆ b₇

When we are taking addresses the following are valid expressions: &A[2] which points to the element just after the array A and &B[0] which points to the beginning of B. So if A and B are adjacent objects, the pointer value of the first expression is exactly the same as for the second. So both represent different semantics but with an abstract address that is the same, the byte address of byte b₀.

So far, so good. Using these pointers with clearly indicated semantics doesn’t pose a problem for aliasing analysis. In particular in our model a pointer value as given above always has a unique provenance. A problem arises only if

• The storage instance of A and B have both been exposed, by a conversion to uintptr_t, say.
• A pointer Λ is synthesized that corresponds to the byte address of b₀ by converting some uintptr_t value back.

Then we are in the dilemma that Λ could have both provenances, that of A or that of B.

Already from the length of the text that is needed to describe the situation you might guess that this is a very rare situation, the easiest remedy against its difficulties is just not to have it in the first place. But in the case that it arises we have to foresee a mechanism that is consistent with the model. Since there is generally no additional information available that could guide the compiler to see which semantics the programmer meant, the semantics are deduced from the usage of the pointer value:

A synthesized pointer value Λ with two possible provenances A or B is assumed to have the one provenance among the two that is consistent with its use in expressions.

That is for example, if we use Λ in

• Λ[0], *Λ or Λ+1 it is assumed that the provenance is the one of B
• Λ[-1] or Λ-1 it is assumed that the provenance is the one of A.

A use as in Λ+0 that has the same value Λ gives no indication of a direction in which we want to step through the array. So it does not fix a choice of one provenance or the other. But Λ+1-1 which resolves to (Λ+1)-1 has the provenance of B and similarly Λ-1+1 ≡ (Λ-1)+1 has the one of A.

So if and when you have to use that marginal constructs, make sure that you give clear indications to the compiler into which of the two storage instance you want to walk.

Takeaway

If you were brave enough to follow this article up to here, you probably deserve some reassuring words such that you are not left alone with a complicated web of choices and interrelationships between parts of your code that are impossible to master. In the contrary, our model provides a very simple method to guarantee sound aliasing analysis by compilers of your every day code:

Do not expose pointer values …

… if you can avoid it. And in most cases you can. In particular avoid

• taking the address of variables (maybe use register to be sure),
• casts of pointer values to and from integers,
• accessing individual bytes of pointer targets (AKA coversion to character pointers),
• conversions of pointer values to any other unrelated pointer type,
• accessing individual bytes of byte-representations of pointer values,
• printing pointer values with %p or by using fwrite,
• using the end address of an array to walk backwards into the array,

and you will be fine.

Obviously, these features are important for C and contribute for a lot of its power. But you should only use them when (and where) you master them in all of their consequences. For example, if in the context of system’s programming you need the XOR trick for your doubly-linked list, that is fine as long as you are aware, that this might cost you some other optimization opportunities. Or if you are debugging your code, printing pointer values with %p can be crucial, but you should make sure that you disable all traces of such printing when you compile for production.

Generally, using more modern features of C will help your compiler to provide you with more efficient and safer executables. For example, in addition to the above

• not using pointer parameters on functions when you are only interested in the value (our recip function is a bad example),
• using const qualification wherever you may,
• not using casts,
• not using integer zero as a null pointer,
• using signed and unsigned integer types consistently,

all contribute to help your C compiler to do at what it’s best: nagging you about things that you might have overlooked.

Martin Uecker has started a new initiative to ensure a better const contract for C2y: change the type of string literals to a const-qualified base type, much as it is already the case in C++. Compilers support this since a very long time; some of them have this as default, some provide command line switches for that model.

Nevertheless, this would be normative change and might be some burden for existing code. So, before doing this and writing papers, it would be good if we had an idea of the impact of such a change in existing code bases. I would be very grateful if we’d receive feedback from you along the lines of

• You have a project that already uses options such as gcc’s -Wwrite-strings (or even a compiler with such a default) to have all string literals const-qualified.
• You have a project and you tested it with such options and introducing this change would be easy. (If so does this change expose some qualification bugs in your code base?)
• You have a project and you tested it with such options, but introducing this permanently into your code base would be a real pain.
• Your project does not care about const and you wouldn’t even know where to begin. If it were introduced in a future version of C, you probably you would have to use the command line to switch such a feature off.

The goal here is not to have an opinion poll or similar; I am not convinced that such polls on some random blog like mine have any particular meaning. I really like to have facts first, not opinions:

• If it is open source, please give a pointer to your project. Otherwise, please describe your project e.g if it is a commercial product of your company or employer. But please, do not share confident information that could get you in trouble.
• Report on your experience with these kind of options for const qualification.
• Don’t speculate about what could happen, restrict yourself to facts.

This week on the C committee mailing list we had a discussion about how C’s types are organized into different categories. At the end I came up with a diagram with that organization. It basically translates the section “6.2.5 Types” of the C23 standard into a graph of inclusions.

Sorry, it is probably a little bit too wide for your phone, but by opening it on a computer screen or a laptop in a separate window and using the zooming feature of your browser you should be able to discover its parts. Annotations give the paragraph number in that clause of the C23 standard.

There is some minor color coding:

• Special cases of additions to certain categories are noted with a red arrow
• Inclusions that are not mentioned explicitly in the standard but that follow from the extent of the underlying set of types are dashed blue.
• Type correspondence where a type or type category is used to define another are dotted lines.
• Narrow integer types that are promoted (usually to int) are in dark green.

C++ seems to finally converge with their contracts proposal, https://wg21.link/p2900. I decided to give it a try and come up with ideas how such a thing would look for C. This is in early stages, not a full proposal yet, and still would need implementation by some of the major compilers.

In particular, the C++ feature is full of sidetracks that I don’t like at all, such as user-defined global handlers and ignorability. But there is a core of ideas, syntax and semantics that I found interesting and that I think should be considered for C. The principal features are

• Contract pre- and postconditions add verifiable conditions that are added to function interfaces. They are checked on entry to a function call and provide knowledge that is valuable for the code following a call.
• If the condition is an integer constant expression, a pre- or postcondition is similar to an invocation of static_assert. If the condition is false, compilation fails. In the following we will not illustrate this aspect to simplify the discourse.
• They have no implications on ABI.
• They can be made composable. If applied rigorously, such conditions can often be deduced from the context. They need not be checked dynamically.
• Besides verified correctness, they offer optimization opportunities for the context of a function call and for the called function itself.

To see how all of this works we first need understand two fundamental primitives, assertions and assumptions. Their syntax looks something like

contract_assert(COND, "some explanatory text");
contract_assume(COND, "some explanatory text");

Here, in both cases COND is some condition that is supposed to be fulfilled.

The first, contract_assert, is quite similar to a feature that we already have in C namely the assert macro. Both test for the condition; if it is fulfilled nothing happens and execution goes on. If it is not fulfilled, a diagnostic is printed to stderr and abort is called to end the execution. The difference to assert is that contract_assert always has to be correct C code. There is no such thing as NDEBUG that pretends the assertion doesn’t even exist.

The other feature, contract_assume, is more subtle and actually quite dangerous if applied carelessly. It is as if defined as

#define contract_assume(COND, ...) do { if (!(COND)) unreachable(); } while (false)

Here unreacheable() is the new macro from C23 (and C++23) that makes the behaviour undefined whenever the branch of the invocation is reached. So in general such a condition would not even be evaluated, but it would be taken for grated that it holds. (It would be evaluated for its side effects if it had any, but you shouldn’t do such things, anyhow.) Take an example

contract_assume(p, "pointer must never be null");
*p = 34.7;

Here we, the programmer, promise that the pointer is never null and the compiler can assume that for the following without checking.

With contract_assert we can give an idea what pre- and postconditions look like from outer space. Take a conventional C function with a declaration in a header file and a definition in a .c file:

// .h header
void* my_malloc(size_t size);

// .c definition
void* my_malloc(size_t size) {
   return malloc(size);
}

This function is admittedly a bit lame. It has just one function call in the body. However, please take this as a simple example for illustration. With contracts, the declaration could look something like

void* my_malloc(size_t size) pre(size) post(r: r);

This declares that the argument to size of a call to my_malloc must always be checked to be non-zero and that the return value, here named r, will always be checked such that it is non-null. An implementation with current tools of that function that takes these conditions into account, but doesn’t reflect them in the declaration part could look as follows

// .h header
inline
void* my_malloc(size_t size) {
   contracts_assert(size, "can't allocate empty buffer");
   defer { contracts_assert(defer_return_value, "allocation failed"; };
   return malloc(size);
}

// .c instantiation
extern void* my_malloc(size_t size);

Putting this as an inline function in the header ensures that the contracts are visible for all callers, and thus, when integrating this into the control flow, the conditions can either be deduced (e.g if we know that the argument is non-zero) or can be used to deduce properties in the caller (e.g the returned value is always non-null).

Note also, that here we use defer to ensure that the assertion triggers regardless on how we leave the function. This is probably not so interesting for this short example. However, if the function has complicated nesting with multiple returns, defer simplifies the process. It helps with code that we want to execute unconditionally on function return.

Changing all functions to inline functions would deviate much from the usual abstraction between interface and implementation in C (and C++). So contracts are an attempt to enrich the interface such that the core of the function stays in its own file and remains known only via the provided interface. A first advantage of pre and post should be obvious. Since it is part of the declaration, it helps people to understand what to expect of a function. It may also assist compilers in the same way. In particular the postcondition can help to take conditions ahead into the context of the caller:

double* p = my_malloc(sizeof(*p));
*p = 34.7;

Here the call to my_malloc, instead of using malloc directly, ensures that the pointer is always checked. A sensible message is printed before the execution aborts, if necessary. For any implementation of postconditions we would want the compiler to deduce that property directly, only from the declaration.

Unfortunately, this simple top level view isn’t very satisfactory:

• A precondition that is hidden in a .c file is only checked when the function has already been entered. In the calling context it might already be known that size is not zero. This information is lost and the assertion has always to be checked at the start of the function.
• If the postcondition is also checked within the function call, we have to come up with a mechanism that propagates the information to the calling context.

To improve these aspects, the following observation about assertions and assumptions is key:

In a series of assertions and assumptions, if they check the same condition without an intermediate change of the execution state, only the first assertion or assumption is necessary. All subsequent assertions or assumptions are redundant.

Take the null pointer check from above. Repeating the assumption clearly does not add any new information for the compiler:

contract_assume(p, "pointer must never be null");
contract_assume(p, "pointer must really not be null");
*p = 34.7;

Similarly, mixing an assertion and an assumption on the same condition does not achieve much:

contract_assert(p, "pointer must never be null");
contract_assume(p, "pointer must really not be null");
*p = 34.7;

Here the compiler has to make sure that the assertion holds, so afterwards the condition can be assumed. If we invert the order:

contract_assume(p, "pointer must never be null");
contract_assert(p, "pointer must really not be null");
*p = 34.7;

When reaching the assertion, the compiler knows that the condition has just been checked. Thus, the whole assertion can be omitted. This observation can help develop an execution model for function with contracts. It fits into C’s current execution model.

• A precondition c on a function f can be replaced by an assertion of c that is executed before any call to f that is combined with an assumption of c at the start of the execution of the function body.
• A postcondition c on a function f can be replaced by an assertion of c at the end of the execution of the function body combined with an assumption of c that is placed after any call to f.

Let’s visit this with our my_malloc example. If we want to execute the precondition before going into the real call of our function we can replace the declaration with the contracts by an inline function:

/* in a .h header */
inline
void* my_malloc(size_t size) {
   contracts_assert(size, "can't allocate empty buffer");
   defer { contracts_assume(defer_return_value, "allocation failed"; };
   return my_malloc_inner(size);
}

That inline function makes the necessary assertions and assumptions. If a call to it is inlined at some point, the compiler sees these and can use the information to check and optimize. Then it tail-calls another function my_malloc_inner that contains the code that otherwise would just have been the definition of my_malloc.

Then, in the .c file, we have to give the implementation of my_malloc_intern and the instantiation of the inline function.

/* in a .c implementation file */
extern void* my_malloc(size_t); // instantiation of the inline function

void* my_malloc_intern(size_t size) {
   contracts_assume(size, "can't allocate empty buffer");
   defer { contracts_assert(defer_return_value, "allocation failed"; };
   // now do everything we need to do
   return malloc(size);
}

This combination of an inline wrapper for the contracts and an auxiliary function for the inner workings of the functions guarantees the requested behaviour:

• the precondition is checked before entry of the auxiliary function
• the precondition is known to hold on entry of the auxiliary function
• the postcondition is checked at the end of the auxiliary function
• the postcondition is known to hold after a call to the auxiliary function.

So this combination has the properties that we want from contracts: the calling context and the called context can deduce the information from the contracts to optimize. But this approach has still a major disadvantage, we have to repeat the conditions within the code of the auxiliary function. This is tedious and error-prone.

The latest release of eĿlipsis provides a mechanism that avoids that code duplication. The contracts are expressed basically as above with an inline function:

/* in an -interface.h header */
inline
void* EXBINDING(my_malloc)(size_t size) {
   DEFER_TYPE(void*);
   ellipsis_precondition(size, "can't allocate empty buffer");
   ellipsis_postcondition(defer_return_value, "allocation failed"; };
   // forward declare auxiliary function
   extern typeof(EXBINDING(my_malloc)) INBINDING(my_malloc);
   // call auxiliary function with same parameters
   return INBINDING(my_malloc)(size);
}

The difference here is that the conditions are expressed with special macros. These macros make it easier to place the right pairs of assertion/assumption inside the inline function. They also help with the implementation. Macros EXBINDING/INBINDING deal with the naming of auxiliary functions. The implementation file then looks very much like the original function definition from which we started.

// -implementation.c definition
void* EXBINDING(my_malloc)(size_t size) {
   return malloc(size);
}

Only the EXBINDING macro is needed to get the name right. This approach expresses all the contracts in the header file. It also leaves the core of the implementation almost unaltered. If you are interested in the details of this approach please refer to the corresponding eĿlipsis documentation.

All of this is unfortunately nothing that can be used easily, yet. A proper interface specification similar and compatible to the one used by C++ would be needed. Nevertheless this is a proof of concept of such an approach for C. The tools we needed are

• defer together with defer_return_value to ease the expression of postconditions
• typeof to ease the forward declaration of functions with the same prototype
• abort to have a defined strategy for termination
• unreachable to express assumptions through control flow
• inline to place conditions into the vision of every user of a function

The new revision of eĿlipsis (20250219) has a lot of cleanups, bugfixes etc, but one thing I’d like to emphasize is a new feature that I’d call meta-quotes in lack for a better idea of a name that implement exemption of tokens from macro replacement. So in C that would interact with translation phase 4.

The idea is to have two new quote characters ⸤ and ⸥ that protect everything between them to be macro expanded.

When, after lexing, the input tokens are filtered by a C-like preprocessor, a nominal (identifier in C speak) that correspond to the name of a macro is expanded. C only has two exceptions from this:

• If the current filtering is the expansion of a macro named ID, other occurrences of that same name ID are not expanded. So preprocessing has no normal recursion.
• If a macro name ID of a functional macro (that is a macro with a parameter list) occurs and that name is not followed by an opening parenthesis (possibly preceded by whitespace) then this token is not expanded, either.

The lack of the possibility to protect other nominals can result in a lot of irritation in contexts where names of identifiers are glued together by means of the token join operator ⨝ (## in C). You’d see things such as the following quite commonly in preprocessor oriented C headers:

#define MYFUNC(F) hurlipurtz ## F

The user then expects the following

    void* MYFUNC(getone)();

to expand to

    void* hurlipurtzgetone();

But in fact it wouldn’t always. If this code is located in a header and another header is included that would define hurlipurtz to expand to schnurtz, the above function declaration would read.

    void* schnurtzgetone();

not at all what was intended. Obviously for the given example this is highly unlikely but the smaller the name components that are used get, the higher the probability of collision. For example a lot of code would use the token _ to glue together two name components, so whenever some smart colleague thinks they should overload that token with something funny, you are in trouble.

To protect against this effect, as an extension, eĿlipsis adds a third method for identifier protection, the keep and peek brackets ⸤ and ⸥. Rewritten with these, the above definition would look

#define MYFUNC(F) ⸤hurlipurtz⸥ ## F

That is, here the funny brackets (BOTTOM LEFT HALF BRACKET and BOTTOM RIGHT HALF BRACKET) protect the nominal hurlipurtz to be expanded when MYFUNC is used. If you are programming in C and are allergic to Unicode characters you could also use the digraphs <| and |> that replace them.

These brackets have the following properties

• They don’t interact with tokenization. That is, the input stream is split up into tokens, first, and possible keep and peek brackets are identified during that process as tokens of their own. They only take their special role in the next phase, filtering, when macros are expanded and directives are interpreted.
• During filtering they apply to arbitrary long sequences of tokens where they protect all nominals that appear in the sequence.
• They can be nested as long keep and peek brackets nest properly.
• If a construct appears between these brackets that is normally ended by an EOL character, the construct extends to the end of the outermost pair of keep and peek brackets.

The latter can be in particular be useful for long macro definition that go over multiple lines

<|
#define SWAP(NAME, T)
 ⸤void⸥ NAME ## ⸤_swap⸥(T* ⸤a⸥, T* ⸤b⸥) {
    T* ⸤tmp⸥ = ⸤a⸥;
    ⸤a⸥ = ⸤b⸥;
    ⸤b⸥ = ⸤tmp⸥;
 }
|>

or equivalently

<|
#define SWAP(NAME, T)
 ⸤void NAME ## _swap(T* a, T* b) {⸥
    ⸤T* tmp = a;⸥
    ⸤a = b;⸥
    ⸤b = tmp;⸥
 ⸤}⸥
|>

or even

<|
#define SWAP(NAME, T)
 <|void NAME ## _swap(T* a, T* b) {
    T* tmp = a;
    a = b;
    b = tmp;
 }|>
|>

Such a definition ensures that the text editor (IDE whatever) basically sees a function, and thus code indentation and highlighting should work as expected. The nested used of keep and peek brackets then also guarantees that the local identifiers a, b and tmp are not accidentally overwritten during macro expansion. So using this macro as SWAP(fine, well) would expand to

void fine_swap(well* a, well* b) {
    well* tmp = a;
    a = b;
    b = tmp;
}

regardless if somebody had defined a macro tmp before or not.

This new release of eĿlipsis uses this new feature extensively in a completely revisited implementation of the defer feature that should be quite robust now and that closely follows the bracketed structure of C (for brackets, braces and parenthesis). Namely there is now a feature (in C) to glue callbacks to this syntactic structure of the code called BLOCKSTATE that automatically tracks integer values that are associated to different levels. For example a macro invocation

__BLOCKSTATE_TST(MINE)

would check if a MINE value is set for the current level of {} (called BRACE_LEVEL) of () (called __PARENTHESIS_DEPTH__) and of [] (called __BRACKET_LEVEL__) and return that value or 0 if there wasn’t one, yet. Similarly there are macros __BLOCKSTATE_SET0, __BLOCKSTATE_INC etc to manipulate state that depends on the current block structure of the program.

The defer implementation uses this feature for example to keep track if the current level of braces is the body of a for-loop or rather a switch to deduce if an unwind of deferred blocks for a continue statement ends on this level of braces or has to go down on the next level.

To implement all of this, the new meta-quote feature has been really helpful. Otherwise I would easily have been lost in the expansion (or not) of all the name components that are glued together in that code.

Last but not least I’d also like to note that the eĿlipsis project is now to be found at Codeberg at

https://codeberg.org/gustedt/ellipsis/

Being previously at the gitlab of my employer, INRIA, has not proven to be much helpful. I am very grateful to the Codeberg initiative for providing such an open and free platform for open source development. If you don’t have an account there yet, go and get it.

The online documentation is still at the INRIA site, so the link hasn’t changed:

https://gustedt.gitlabpages.inria.fr/ellipsis/

I have already talked several times about defer, the new feature that hopefully will make it into a future version of C. With this post I will concentrate on the here and now: how to apply that lifesaving feature with existing tools and compilers.

After briefly discussing the defer feature itself, again, I will show you a first implementation with gcc extensions, a second with C++ standard features and then I will discuss a new proposal with for defer that has a syntax that is a tiny bit more constraining than what you may have seen so far.

The defer feature itself

To remind you what defer is supposed to do, here is some toy code that uses three resources:

{                             // anchor block
    void* p = malloc(25);
    defer { free(p); };          // 1st defer

    mtx_lock(&mut);
    defer { mtx_unlock(&mut); }; // 2nd defer

    static uint64_t critical = 0;
    critical++;
    defer { critical--; };       // 3rd defer

    while (something) cnd_wait(&cond, &mut);
    ... use p under protection of mut ...
}

This code with three so-called deferred blocks is equivalent to the following

{
    void* p = malloc(25);

    mtx_lock(&mut);

    static uint64_t critical = 0;
    critical++;

    while (something) cnd_wait(&cond, &mut);
    ... use p under protection of mut ...


    { critical--;       } // from 3rd defer
    { mtx_unlock(&mut); } // from 2nd defer
    { free(p);          } // from 1st defer
}

only that the deferred blocks are even executed when the anchor block is left by a jump statement (such as a break, continue, return or even goto) that would be nested inside complicated if/else conditionals. Thus using defer here ensures that

• p, something and critical are only used under the protection of the mutex mut,
• critical then holds the number of threads that are currently within that anchor block
• mut never remains locked whenever the anchor block is left.
• *p is deallocated whenever the anchor block is left.

Implementing defer with gcc

With a minimal macro wrapper this feature works out of the box in gcc since at least two decades. Written with C23’s attribute feature the inner macro looks as simple as the following:

#define __DEFER__(F, V)      \
  auto void F(int*);         \
  [[gnu::cleanup(F)]] int V; \
  auto void F(int*)

Here the three lines of the macro are as follows:

• auto void F(int*); forward-declares a nested (local) function F.
• [[gnu::cleanup(F)]] int V; establishes this function as a cleanup handler of an auxiliary variable V
• The second auto void F(int*) then starts the definition of the local function which is completed with the user’s compound statement as the function body.

For this to work we have to provide unique names F and V such that several defer blocks may appear within the same anchor block. This is ensured by another very common extension __COUNTER__

#define defer __DEFER(__COUNTER__)
#define __DEFER(N) __DEFER_(N)
#define __DEFER_(N) __DEFER__(__DEFER_FUNCTION_ ## N, __DEFER_VARIABLE_ ## N)

That is basically it, a straight application of the [[gnu::cleanup]] feature that

• avoids the need for inventing safe identifiers,
• avoids the definition of an one-shot function with internal or external linkage far away from its use,
• integrates well and quite efficiently with the existing compiler infrastructure.

Indeed, when adding a bit more magic (such as [[gnu::always_inline]]) the assembly that is produced is very efficient and avoids function calls, trampolines and indirections. (See also Omar Anson’s blog entry on how efficient the cleanup attribute seems to be implemented in gcc.)

If you don’t like or have the C23 attribute syntax yet, you should easily be able to use gcc’s legacy __attribute__((...)) syntax without problems.

Implementing defer with C++

Yes! I think it would even make sense for C++, but what do I know.

At least the following implementation shows that the feature fits directly into C++’s model of binding actions to a scope. In fact, implementing the defer feature with the properties as desired in C++ is a student excercise. It can be done with a template class and lambdas.

template<typename T>
struct __df_st  : T {
  [[gnu::always_inline]]
  inline
  __df_st(T g) : T(g) {
    // empty
  }
  [[gnu::always_inline]]
  inline
  ~__df_st() {
    T::operator()();
  }
};

#define __DEFER__(V)  __df_st const V = [&](void)->void

Lambda expressions have a unique type for each expression. Such a lambda expression is taken here as a template parameter to the constructor __df_st<T>::__df_st(T). The destructor __df_st<T>::~__df_st() of the variable then invokes the lambda when V leaves its scope. The [&] in

[&](void)->void { some code }

ensures that all outer variables are fully accessible at the point of execution of the lambda. Therefore, such an implementation provides the full functionality that we need for our first example from above.

Note, that the __COUNTER__ pseudo-macro is also quite commonly implemented by C++ compilers and is now even proposed as an addition to C++26. Thus with a similar macro as for gcc above we can implement the defer macro itself:

#define defer __DEFER(__COUNTER__)
#define __DEFER(N) __DEFER_(N)
#define __DEFER_(N) __DEFER__(__DEFER_VARIABLE_ ## N)

A note on the syntax for defer

As we have seen above existing implementations (let’s also count the one for C++) use quite different features under the hood:

One uses a function declaration where a compound statement of the user code as in

    defer { mtx_unlock(&mut); };

ends up being a function body. Here the terminating ; is superfluous, but doesn’t hurt much, either.

The other declares a expression that also needs {} to limit the user code and a terminating ; because underneath it is an object declaration.

I think we should combine these different syntax requirements, such that implementors (or library providers) may easily start providing this feature with what they have. I have recently made a new proposal to the C standards committee in that sense:

Even simpler defer for direct integration, N3434

It has the advantage, I think, that it is much easier and more direct that previous proposals and that it combines the possibility of implementing the feature in the language or in the library. Therefore the defer features is proposed as a “block item” in a compound statement (its anchor) and then defined via the syntax rules

defer-block:

defer deferred-block ;

deferred-block:

compound-statement

The C23 edition of Modern C is now available for free download from

https://hal.inria.fr/hal-02383654

And as before a dedicated page for the book may be found at

https://gustedt.gitlabpages.inria.fr/modern-c/

where you also may find a link to download the code examples that come with the book.

This new edition has been the occasion to overhaul the presentation in many places, but its main purpose is the update to the new C standard, C23. The goal was to publish this new edition of Modern C at the same time as the new C standard goes through the procedure of ISO publication. The closest approximation of the contents of the new standard in a publically available document can be found here. New releases of major compilers already implement most of the new features that it brings.

Among the most noticeable changes and additions that we handle are those for integers: there are new bit-precise types coined _BitInt(N), new C library headers (for arithmetic with overflow check) and (for bit manipulation), possibilities for 128 bit types on modern architectures, and substantial improvements for enumeration types. Other new concepts in C23 include a nullptr constant and its underlying type, syntactic annotation with attributes, more tools for type generic programming such as type inference with auto and typeof, default initialization with {}, even for variable length arrays, and constexpr for named constants of any type. Furthermore, new material has been added, discussing compound expressions and lambdas, so-called “internationalization”, a comprehensive approach for program failure.

Also added has been an appendix and a temporary include header for an easy transition to C23 on existing platforms, that will allow you to start off with C23 right away.

Manning’s early access program (MEAP) for the new edition is still open at

https://www.manning.com/books/modern-c-third-edition

Unfortunately they were not yet able to tell me when their version of the C23 edition will finally be published.