rdist

May 3, 2010

Why buffer overflow exploitation took so long to mature

Filed under: Hacking,Network,Security — Nate Lawson @ 6:00 am

I think the history of buffer overflow exploits is interesting because of how long it took for techniques to mature. About 16 years passed from awareness to first public exploitation, and then 8 more years from that until they were commonly exploited. Programmers were aware of this class of flaw but did little to avoid them for 24 years. But why?

Executing code via a buffer overflow was published at least as early as 1972 (pdf). It’s quite likely this was common knowledge before then, as overlay techniques (loading one segment over another during execution) were often used to save RAM. The first public malicious exploitation of a buffer overflow was by the 1988 Internet worm.

The Morris worm exploited a flaw in the finger daemon, which used gets() to read the username from the client into a buffer on the stack. Its shellcode was only targeted at 4.3 BSD Unix on the VAX CPU, and it used a return address overwrite to gain control. Interestingly enough, the worm did not have shellcode for SunOS even though the Motorola 68k CPUs would have been easy to exploit, and Sun’s fingerd was vulnerable to the same flaw.

Public exploitation of overflows took a 7-year hiatus, even with the publication of a seminal 1990 paper on fuzz testing. Everyone seemed to know overflows were present and theoretically exploitable, but no one seemed to be exploiting them or worried about fixing them.

Then in February 1995, Thomas Lopatic published a stack overflow exploit for NCSA httpd on HP-UX. This was an excellent piece of work, but on an obscure OS and CPU. (By the way, Thomas was also known for his work on a loader for Amiga games and later for analyzing the Windows XP activation scheme).

However, buffer overflow exploits were still few and far between. In August, 8lgm published an advisory for syslog() on SunOS SPARC, but no public exploit. In December, the splitvt exploit for Linux x86 was published. However, a month later, some people were still wondering how it worked.

In November 1996, Aleph1 published “Smashing the Stack for Fun and Profit“. This important article described in detail the evolution of a stack overflow exploit. Though I don’t know the exact history, Aleph1’s x86 shellcode is nearly identical to the splitvt exploit, so perhaps his method was derived from it. The article also provided shellcode for SunOS and Solaris on SPARC, which was an important advance since such machines were still more “interesting” than x86 systems.

After this paper, numerous stack overflows were published. Research on both sides advanced rapidly, with new techniques such as heap overflow exploitation and stack integrity protection. So why had research in this area taken so long to reach this inflection point of rapid growth?

In the next article, I discuss various factors that collided in 1996, creating the buffer overflow arms race that continues to this day.

4 Comments

  1. Around ~1983 C64 commercial program loads were smashing the (page 1) stack to take control of the machine (and probably the technique was in use on the Apple II much earlier). Some might argue this was just saving the user from typing “run” and therefore not “malicious”. But I would say the purpose often included the intent–similar to that behind the concurrent use of illegal opcodes–of attempting to prevent the user/program-owner from readily saving, modifying or learning from the program. I don’t doubt some members of the FSF would indeed consider that “malicious”. YMMV.

    Comment by Dan Hite — May 3, 2010 @ 8:49 pm

  2. In the splitvt exploit, the reason for the variable names of eggplant, egg, egg2, etc, which led to early calling of the shellcode as ‘egg’, was due to DaveG’s love for eggplants.

    Comment by Danny Dulai — August 22, 2010 @ 11:57 pm

    • That’s hilarious. I like to remind people he once had his own cooking show.

      Comment by Nate Lawson — August 26, 2010 @ 11:29 am

  3. I remember seeing and writing exploits for buffer overflows for SPARC around the age of 16-17, so that was 1994. Took me an entire weekend to learn SPARC assembly and write my first one, so hard was it back then ;)

    Comment by RPW — August 25, 2010 @ 6:51 pm


RSS feed for comments on this post.

Blog at WordPress.com.