rdist

October 5, 2007

C64 screen memory and anti-debugging

Filed under: C64,Hacking,Security,Software protection — Nate Lawson @ 5:00 am

I think it’s fun to stir your creativity periodically by analyzing old software protection schemes. I prefer the C64 because emulators are widely available, disks are cheap and easy to import, and it’s the system I became most familiar with as a kid.

One interesting anti-debugging trick was to load the protection code into screen memory. Just like on the PC, the data on your screen is just a series of values stored in memory accessible to the main CPU. On the C64, screen memory typically was located at 0x400 – 0x7FF. Data could be loaded into this region by setting the block addresses in the file’s directory entry (very simple version of shared library load address) or by explicitly storing it at that address using the serial load routines.

To keep users from seeing garbage, the foreground and background colors were set to be the same. If you tried to break into a debugger, the prompt would usually overwrite the protection code. This could be worked around by relocating actual screen memory (by reprogramming the VIC-II chip) or by manually loading the code at a different address and disassembling it.

This is an example of anti-debugging based on utilization of shared resources. The logic is that a debugger needs to use the screen to run, so if the protection is using that resource also, the attacker will disrupt the system by activating the debugger. It is usually much more effective to use up a shared resource than to just check for signs that a debugger is present, an approach that is still important today.

6 Comments

  1. What is an example of this technique in a modern product?

    Comment by Matt — October 5, 2007 @ 9:12 am

  2. Using screen memory? None that I know of. But the general approach of utilizing shared resources is still very relevant.

    For example, storing some of your code in the RAM page used for the 1394 debugger stub under Windows means attaching a debugger would overwrite that code. This is more powerful than checking the KdDebuggerNotPresent variable. There are many other examples.

    Comment by Nate Lawson — October 5, 2007 @ 4:36 pm

  3. How can you put code in the RAM page used for the 1394 debugger stub with an ordinary Windows application? Doesn’t that require a kernel mode driver or similar?

    I’ve enjoyed reading your blog for a while and I think it’s very interesting, although (intentionally?) vague on the details. Maybe that’s a good thing, at least it makes my brain hurt trying to figure out how to actually implement any of your suggestions. Maybe I just look at it from the wrong angle, I read your blog from an independent shareware author’s view.

    Comment by Andreas — October 22, 2007 @ 11:10 pm

  4. Hi Andreas. I am not intentionally leaving out details, I just happen to write from my own experience and perspective. So I may leave out things which I forgot not everyone assumes. :)

    In this case, most of my work is kernel-side. Even when reversing a usermode program, I like to use a kernel debugger. So when I talk about anti-debugging, I sometimes assume the vantage point is kernel mode. Most of the time the general technique is applicable to both.

    Comment by Nate Lawson — October 23, 2007 @ 9:19 pm

  5. Hi Nate, I’m pretty new at cracking/anti-cracking techniques, so I guess I lack some of the fundamentals. Finding your blog was a huge boost for my knowledge in the field. Since I’ve never been a cracker myself I find it hard to think like one. I simply assume the cracker to be omnipotent, but then all my efforts seem to be in vain and I succumb to disillusion.

    It sounds very attractive to store some code (or a crypto-key?) in a shared resource like that, but the more I think about it the more I see the problems prop up. What if two different programs use the same shared resource for their software protection?

    Thanks for writing a great blog!

    Comment by Andreas — October 23, 2007 @ 10:31 pm

  6. Thanks. Before I respond, let me give you a challenge in return. How would you translate this general technique (using a shared resource for anti-debugging) to usermode?

    For other programs concurrently using the same technique, there is no easy solution. Since the whole goal is to avoid shared use of the resource, this is how it’s supposed to work.

    First, you can often assume mostly dedicated access (i.e., playing a game). Asking the user not to play two games at once is usually reasonable. Second, if you control both apps that do the sharing, there are cryptographic ways of ensuring that the attacker isn’t the other party. Third, the most common response mechanism for this approach when it’s possible someone might have a legitimate purpose for using the resource is to fail obviously and softly. An error message to the user describing the problem and how to correct it is a good idea. As always, the response should be far instruction-wise from the check that detects the problem.

    Comment by Nate Lawson — October 25, 2007 @ 10:49 pm


RSS feed for comments on this post.

Blog at WordPress.com.