I’ve always thought disk encryption without integrity protection is worse from a reliability standpoint. You want an accidental bitflip in a sector to be reported to an upper layer (like a RAID mirror) so it can fail over to known good data from another source. CBC alone turns a bitflip into 16 extra bytes of garbage while still not detecting the error. So it makes things worse by itself.

As for the checksum-insertion, Microsoft’s Phoenix compiler project (which allows you to extend the compiler with user-written add-ons) looks like one way of doing this. The downside is that it’s still a bit of a fixer-upper. The closest I’ve seen in the OSS world is Treehydra, a gcc plugin that provides Javascript access to the gcc AST, but that’s read-only access to help in writing static analysis tools rather than a user-written compiler extension capability. Now all we need is (handwave) someone else to write the necessary plugins…

One thing I keep coming back to is the idea that software protection principles will eventually be deployed in every compiler. Just like stack cookies are expected to be enabled in production software today, future compilers will automatically insert compile-time-randomized hash integrity checks for both code and data. This will be great for reliabilty as well, since memory corruption errors (even self-induced, not due to an attacker) will be more obvious.

I’m interested in seeing research in this area because, as you point out, it’s difficult to do in an automated way since you have to explicitly tell the compiler how you will be treating struct members.

One thing that I haven’t seen discussed much is robustness principles for software that go beyond the basic “check for integer overflows” and the like. For example one defence mechanism that MS adopted in recent versions of the Windows allocator is to check, before freeing (unlinking) a memory block, that current->prev == prev->next and current->next == next->prev, to prevent common heap-overflow attacks. This is fairly easy to add to any linked-list manipulation code, but I don’t know of any list of design patterns that can be applied in the same way as, say, the CERT secure coding guidelines.

One particularly challenging issue is how to sanity-check the contents of a large structure containing state information. Say you have an SSL implementation and want to protect it against control-flow attacks (there was an unpublished attack on an SSH implementation a few years ago, for example, that allowed you to bypass the sign-in authentication by overflowing the password field so that it overwrote the nearby ‘is-authenticated’ field with the characters from the end of the password). Lets say your SSL implementation holds all your session state in (say) a ‘struct SSL’ containing a mix of mutable and immutable (after the session is established) fields. Moving the fields around to make the mutable and immutable ones adjacent isn’t really an option because you’d end up with a random mix of unrelated variables (in other words you’d end up with an ‘unstruct SSL’). How do you build a means of checking for unexpected modifications without moving fields around? You end up with a problem that’s a bit like the AH header checks in IPsec, but with many fields present in the struct and a far more complex field structure it’s much harder to do.

(This is an open question, I’d be interested in any comments on how to solve it).

Dave has a great point. As the software stack becomes harder and harder to break attackers will look for issues in the hardware itself. If the software is checking the hardware results attacks based on a hardware flaw could be detected.