Shellcoding and exploits

What is a memory error?

Memory access errors describe memory accesses that, although permitted by a program, were not intended by the programmer. ¹

Memory access errors are often defined ⁹ as:

• buffer overflow
• null pointer dereference
• use after free
• use of uninitialized memory
• illegal free

The software is memory-safe if these errors never occur.

There are usually two main reasons why dangerous memory bugs are possible.

• The software takes user-defined input
• This input is neither validated nor sanitized, so the program flow can be controlled with the input in ways that were originally unintended

This input validation and sanitization is one of the major challenges in software development. You must ensure that every unintended effect of the user-defined input is either prevented or handled.

You want the user to provide a name that is 15 characters long at maximum. What if they provide 20 characters???

Buffer overflows

If your program does not handle input longer than 15 characters as in the previous example, a so-called buffer overflow can happen — as long as the programming language does not add a boundary check automatically.

This error is usually the most dangerous type. MITRE’s top weakness of 2023 (out-of-bounds write) ⁷ belongs to this category.

To understand why, we need to understand how a computer works at the stack level and what the principles of programming languages have to do with it.

The fundamental philosophy of C programming is to “trust the programmer”. Do not prevent the programmer from doing what needs to be done. The programmer has ultimate control, but also ultimate responsibility. This means that they must also use memory correctly.

In the naive example below, the compiler reserves 15 bytes of stack space for the name variable.

This means that a name of at most 14 characters (plus the null terminator \x00) can fit into this buffer. The programmer should know that the null terminator also takes up space.

#include <stdio.h>

int main() {
    char name[15];

    printf("Please enter your name: ");
    scanf("%s", name);

    printf("Hello, %s!\n", name);
    return 0;
}

Since the compiler trusts the programmer, the program only does what it is programmed to do; in this case it does not check the boundaries of the buffer.

If the end user provides input longer than 14 characters, the buffer overflows and writes into the memory area that was not reserved for it.

In practice, a software buffer overflow means that the space reserved for the data is insufficient for the data being stored.

“Buffer overflows are Mother Nature’s little reminder of that law of physics that says: if you try to put more stuff into a container than it can hold, you’re going to make a mess.” ¹⁰

Conversely, a buffer over-read means that a read operation may read more than it should.

We mainly focus on stack buffer overflows in this exercise.

If you are curious about how memory in the heap works, take a look at this page ¹¹.

Understanding the stack

The computer stack is like a stack of books.

1. You can only add (push) or remove (pop) a book at the top (also known as FILO: first in, last out).
2. It’s used to keep track of operations like function calls: when a function starts, its details are added (pushed) to the stack, and when it ends, they are removed (popped).
3. If the pile of books grows past the edge of the table, the whole pile collapses. Something similar happens in a program when the stack grows past its limit — that is a stack overflow. A stack buffer overflow is a different (and more interesting) bug: a single buffer on the stack is written past its own end, which is the vulnerability we are after in this exercise.

When an application runs, it uses the stack and registers to manage the program’s execution flow. The stack is split into frames, each holding the data of a function that has not yet returned. A frame stores the function’s arguments, its local variables, the return address, and more. For instance, a program with three nested function calls generates three stack frames.

Below is a simplified example of the stack of a 32-bit program, where funcA() is called first and then calls funcB().

Dangers of the overflow

While the stack grows towards lower memory addresses, an overflowing local variable writes towards higher memory addresses. See the illustration below.

|---------------------|
| Return Address      |  <-- Higher Memory Address
|---------------------|
| Saved Base Pointer  |
|---------------------|
| Local Variable 1    |
|---------------------|
| Array (e.g., char)  |  <-- End of local array
| (Variable 2)|
|                     |
|---------------------|  <-- Start of local array
| Local Variable 2    |
|---------------------|
| ...                 |  <-- Stack Pointer (Lower Memory Address)
|---------------------|

When the data written into the array (Variable 1) exceeds the space allocated for it, it overwrites the adjacent memory regions, which are exactly the ones that control the program’s execution flow!

If an attacker successfully overwrites the return address, they can dictate where the program resumes execution next. If the manipulated return address points to a location containing malicious instructions, the program will unwittingly execute that code.

In earlier eras, many compilers lacked mechanisms to detect or prevent such overflows. Consequently, these vulnerabilities have sometimes led to arbitrary code execution.

For more information, read the section 2.3, “Stack buffer overflows”, in “Low-Level Software Security for Compiler Developers” ¹.

Shellcoding

The term “shellcoding” comes from the scenario in which these memory bugs are exploited in such a way that they end up opening the computer’s shell.

Manipulating the execution flow of a vulnerable program can potentially result in privilege escalation. A vulnerable program running with system-level privileges might unintentionally run arbitrary code with those elevated rights. Historically, exploiting a setuid program to launch a shell gave the attacker a shell with that program’s elevated permissions. Modern UNIX systems make this harder — the kernel ignores the setuid bit on interpreted scripts, and hardening measures such as privilege dropping and nosuid mounts reduce the number of useful targets — but the setuid bit is still honoured on binaries, so the risk has not disappeared.

Acquiring shell access this way usually leads to full control of the system, which is why spawning a shell is one of the most common goals of attackers.