System Call Policies and systrace

Recently I’ve been looking at a little-known utility called ‘systrace‘, which in theory (I’ll come to that) protects Linux boxes against most exploits and privilege escalation. Basically it controls and logs user-space access to the kernel resources. This isn’t to be confused with the Android OS diagnostic tool of that name.

System Calls
The user and kernel spaces need a method of communicating with each other, especially if a program needs to interact with the harware for memory management, writing to disk, sending data to a network interface, etc., and this is done through ‘system calls’. This is vaguely analogous to functions or .NET objects a program uses to handle whatever operation. In fact, a function itself can contain a system call, so it becomes a ‘system call wrapper’. I’ve yet to figure out how exactly systrace replaces those in glibc.

How is this related to security? The kernel, being the operating system, runs at the highest privilege level, which means a malicious program or exploit could also manipulate it through system calls. Malicous code might also find its way into applications the user trusts, since nobody has time to inspect all the code in every application they use. Worse still, the entire application might be run with root privileges, as happens when launched with the sudo command.

systrace as a Solution
Niels Provos, at the University of Michigan, presented a solution to this in his paper ‘Improving Host Security with System Call Policies‘ (download here). The literature can be a little hard to follow even for an experienced Linux user, and I’ve interpreted it the best I can.
Instead of launching a whole program with root privileges, Provos suggests using ‘system call interposition’ (interception, basically) to control which system calls are allowed, and systrace will permit, deny or ask (the user), depending on whatever policies are set. According to the ONLamp.com page, anything that’s denied by systrace will be recorded in syslog, which I assume is the file in /var/log/syslog.
For performance reasons, the permit and deny rules are enforced in kernel space, while the ask rule causes a user space program to wait for input from the user.

Putting all this together, we can see that systrace actually performs three functions:
* Policy enforcement
* Intrusion detection
* Automated privilege elevation

Intrusion Detection
As we’ve seen, anything denied by systrace is logged, and this is where the intrusion detection bit comes in. This has obvious advantages over perimeter-based IDS, as it records actions made locally as well as remotely, and could potentially reveal why those actions were made. The downside is setting this up across multiple hosts could be a pain in the ass, even if the syslog data were somehow aggregated.

File Permissions vs. systrace
Another advantage systrace might have is better access control than UNIX file permissions. Although the latter gives granular control over what files users read, write and execute, it must be done meticulously in order to be totally effective. There are just too many files. This was the primary reason Provos decided to use something that enforced controls at the system call level.

Disadvantages and Vulnerabilities in systrace
But is systrace really worth installing? Updates and changes have been occasional, with the latest made in 2009. It also has vulnerabilities, but even then it would make exploits much harder overall. Perhaps the main reason it’s not commonly used is it’s been superseded by SELinux and AppArmor, which apply Mandatory Access Controls to the kernel itself.

As I understand it, the main vulnerability in systrace is a kind of ‘race condition’ or timing attack, where a malicious program changes a system call just after it’s permitted, although Provos did anticipate this in his paper. Whether that’s likely to happen inpractice is anyone’s guess, but it’s possible.

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s