There have been a few new developments regarding the recent PS3 USB exploit. Working with impressive speed, Mathieulh and other developers have released an open-source version of the exploit called PS Groove. A much more detailed analysis of PS Jailbreak was also posted, although it is still not completely clear how the exploit works.
The PS Groove exploit uses an AT90USB board with the excellent LUFA library as I had expected. (By the way, send some donations to Dean Camera if you use that library. He’s a generous developer.) It attaches the proper config descriptors in the right order but contains a different payload. It will also allow you to disconnect the USB device after the exploit is complete.
Now that more details are public, the exploit is quite impressive. It is a heap overflow, not stack overflow as Gamefreax had suggested. Also, I was right that they had misread the descriptor lengths (0x4D vs. 0xAD).
The exploit involves using various config/interface descriptors to align shellcode on the heap. Then through some still-unknown mechanism, a heap overflow gives a user-controllable function pointer, which is later called after free(). The bug appears to be related to how the PS3 enumerates Sony’s internal test JIG device. This device may be probed by a different portion of the kernel, which trusts the device’s USB descriptors more.