Michael previously gave a nice talk summarizing the security flaws in the original Xbox.
The CPU itself supports hashing and/or encryption of physical pages, based on flags set in the upper word of the 64-bit virtual address. They talk about how Felix was able to leapfrog off shader-based DMA to write to an unencrypted register save state structure, jumping through a syscall gate (sorta like return-to-libc) that was improperly validated by the hypervisor. The end result was arbitrary code execution in the context of the hypervisor. Quite impressive.
I’ve always wondered how different security features like encrypted RAM that have long been present in smart cards would take to “trickle-up” to the more complex platforms like game consoles. While the Xbox 360 security is much better than the original Xbox, it seems like the big-systems people are reinventing techniques already tested and worked out in the microcontroller world.
For example, the 360 was vulnerable to a timing attack, where an internal secret key can be guessed by timing how long it takes to validate the submitter’s HMAC. I’d be extremely surprised if any mainstream smart card were vulnerable to such a well-known legacy bug.
I have yet to see anyone publish information about applying power or RF-based side channel analysis to a game console, despite smart cards adding countermeasures to these almost 10 years ago. Even earlier attacks on encrypted RAM have still not been attempted on modern systems.
These attacks probably haven’t been needed yet since software bugs were still present. However, the push by game consoles and cellphone manufacturers to increase their resistance to software attacks means it won’t be long before side-channel resistance becomes a must-have feature. It will be interesting to see how long it takes big-system manufacturers to add countermeasures and whether they’ll choose to learn from the hard lessons we have seen in the smart card world.