Introduction

In this article I will demonstrate an interesting way to use the inotify API in Linux to discreetly record information about newly-created processes as a low-privileged user. The technique used to record process information that is featured in this article is the same technique used by the popular offensive security tool pspy. By reading this article, you will be exposed to the inotify API in Linux and will effectively understand how the pspy tool works. Please note that this article is primarily written from an attacker’s perspective. As a root user, there are much more convenient ways to monitor the system for new processes (e.g. eBPF). Before I get into the meat of this article, I want to first discuss a little bit about processes in Linux.

Processes

Whenever a process is created in Linux, a directory for this process is also created at /proc/[pid], where [pid] is the process ID of the process. This directory holds information about the process itself, and is readable by anyone, by default. I repeat, readable by anyone. In other words, on a default installation of most Linux distributions, any user can obtain information about any other process. In fact, traditional Unix and Linux system administration tools such as ps and top access the directory /proc to retrieve and print information about processes. For example, to find out which command was executed to start process number 1, execute the following command in a Linux shell:

cat /proc/1/cmdline

Example output:

/sbin/init

The /proc/[pid] directory for each process is only available while the process is executing. Once the process has finished execution, the associated /proc/[pid] directory will no longer be available. This is very important to understand when using traditional tools like ps and top. Upon executing one of these tools, they gather process information by reading files under each /proc/[pid] directory for each process that is executing. In other words, they take a snapshot of processes that are currently executing. This means they can only gather information about processes that are already in the state of execution. For the majority of system administration tasks, this is probably sufficient. However, if a process executes and quickly exits, it’s likely that one of these tools will not catch it. You will have no idea these short-lived processes ever executed at all. This is where traditional tools fall short. In the rest of the article I will describe two possible ways to circumvent this limitation and record information about new processes as a low-privileged user.

In this introduction I have briefly covered processes in Linux, the traditional tools used to gather information about processes, as well as one of the shortcomings of these tools. In the next section, I will show a resource-intensive way to record information about newly-created processes. Following that, I’ll introduce the inotify API and describe how it can be used to achieve the same results while using substantially less system resources. Finally, I will conclude this article by demonstrating why it’s bad practice to pass clear-text credentials on the command line in Linux.

Outline

• Continuously monitor /proc.
• Use inotify API.
• Attack demonstration.

Continuously Monitor /proc

I’ll start by discussing the most obvious way to record information about each newly created process in Linux. Since a new directory is created for each new process at /proc/[pid], why not just continuously list subdirectories of /proc in search of these new process directories? Here is a Python script I created that does exactly that:

#!/usr/bin/env python3
from datetime import datetime
import os, signal, sys

# Exit on CTRL+C.
def signal_handler(signal, frame):
    sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

def _main():
    # Latest process ID.
    pid = 0

    # List of process directories under /proc.
    proc_list = []

    # Continously check for new process.
    while(True):

        # Retrieve ordered list of process directories under /proc.
        proc_list = os.listdir("/proc")
        proc_list = [val for val in proc_list if val.isdigit()]

        # Check for new processes.
        for i, dir in enumerate(proc_list):
            dir = int(dir)

            # New process.
            if dir > pid:
                
                # Retrieve information about new process.
                try:
                    proc_path = '/proc/' + proc_list[i]

                    # Effective user ID.
                    with open(f'{proc_path}/status') as f:
                        for line in f:
                            if line.startswith('Uid:'):
                                uid = int(line.split()[1])

                    # Command line.
                    with open(f'{proc_path}/cmdline') as f:
                        cmdline = f.read().replace('\0', ' ')

                # Process may have died.
                except:
                    pass
                
                # Print new process information with timestamp.
                time = datetime.now()
                time = time.strftime("%Y-%m-%d %H:%M:%S")
                print(f'{time} UID={uid}\tPID={proc_list[i]}\t| {cmdline}')

                # Linux assigns process IDs sequentially.
                pid = dir

if __name__ == '__main__':
    _main()

This script can be summed up in the following steps:

• Get ordered list of process directories under /proc.
• Compare each directory name to the last-recorded process ID.
• If the directory name is greater than, it’s a new process.

Linux assigns process IDs sequentially starting from zero, up to a maximum number specified in the file /proc/sys/kernel/pid_max. To record new processes, you only need to remember one process ID at any given time. The next process ID which is greater than the most recently recorded process ID is a new process. In my experiments I have never reached this max process ID number. To keep things simple, I did not include a check feature for this number in my scripts.

Less talk, more do! In the following demonstration, I have created some cron jobs to execute the following command every 10 seconds:

ping -c 1 lo

Here I will let the script execute and see if it’s able to catch these cron jobs:

The script successfully prints the command line of each cron job as they execute. At first glance, this seems to be sufficient. Information about new processes is printed as the processes are created and executed. Why even bother with the inotify API? Well actually, if we take a look at system resource usage while this script is executing, it becomes obvious why continuously listing subdirectories of /proc in an endless loop might not be a wise choice as an attacker.

Here is the script running again on the left, but this time I am monitoring system resource usage via top on the right:

If you look just under the white line in the output of top on the right, you’ll see that the script is using almost 100% of the CPU time. This would look pretty suspicious to anyone that happens to be monitoring the system resource usage while the script is executing. How can the same results be achieved while using substantially less system resources? Introducing the inotify API.

Use inotify API

According to Wikipedia:

inotify monitors changes to the filesystem, and reports those changes to applications.

What does this have to do with processes? It turns out that when processes are executing in Linux-based operating systems, they often access other directories in the filesystem during their execution. Using strace, I can see which system calls are called by a process, and thus see which files (if any) the process accesses. I’ll use it on the program whoami:

strace whoami

The command prints a ton of output, but I only care about lines which indicate the whoami program accessed other files:

execve("/usr/bin/whoami", ["whoami"], 0x7fff5aba2260 /* 21 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
read(3, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 553
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3

To my surprise, this program which simply prints my username has accessed several different files and directories throughout the filesystem. Using strace I discover that even the simplest programs often tend to access various files throughout the filesystem. By using the inotify API in my script, I can be notified each time a process accesses one of these files, and subsequently scan the /proc directory for new process directories.

Before using the inotify API in my script, I want to show how it can be used to monitor a directory for changes in the filesystem (e.g. read, write, delete). Here is a simple script that uses the inotify API to monitor the directory /tmp for filesystem activity:

#!/usr/bin/env python3
import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()

    # Monitor /tmp for filesystem events.
    i.add_watch('/tmp')

    # Do something on each filesystem event.
    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event

        # Print each event.
        print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format(
              path, filename, type_names))

if __name__ == '__main__':
    _main()

Here the script executes on the left, and I play around in the /tmp directory on the right:

As you can see, each time I list, create, read or delete files in the /tmp directory, the script prints information about the event. If you want to learn more about these different events and how they work, please see inotify(7). That is a big topic and is out of scope for this article, but I hope this article may encourage you to do your own research about the technical details of the inotify API if you’re interested. For the purpose of this article, I only care that I can use inotify to monitor a filesystem for activity.

Now that I have given an example of how the inotify API can be used to monitor a filesystem for activity, I’ll add it to the script from earlier. I want to monitor the following directories for activity because processes often seem to access these directories:

/usr
/tmp
/etc
/home
/var
/opt

Disclaimer: I may or may not have copied this list from the pspy repository.

Any time a process accesses a file in one of these directories, I want the script to immediately check /proc for new process directories. It’s pretty much the same as the first script, except now it only checks for new process directories under /proc when there is filesystem activity detected in one of the aforementioned directories. Here is the updated script:

#!/usr/bin/env python3
import os, signal, sys
from datetime import datetime
import inotify.adapters

# Exit on CTRL+C.
def signal_handler(signal, frame):
    sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

def _main():
    # Latest process ID.
    pid = 0

    # List of process directories under /proc.
    proc_list = []

    # Use inotify API to listen for filesystem activity.
    i = inotify.adapters.Inotify()

    # Directories to monitor for filesystem activity.
    dirs = ['/usr', '/tmp', '/etc', '/home', '/var', '/opt']
    
    # Begin watching directories for events.
    for dir in dirs:
        i.add_watch(dir)

    # Check for new process on each filesystem event.
    for event in i.event_gen(yield_nones=False):

        # List process directories under /proc.
        proc_list = os.listdir("/proc")
        proc_list = [val for val in proc_list if val.isdigit()]

        # Check for new processes.
        for i, dir in enumerate(proc_list):
            dir = int(dir)

            # New process.
            if dir > pid:
                
                # Retrieve information about new process.
                try:
                    proc_path = '/proc/' + proc_list[i]

                    # Effective user ID.
                    with open(f'{proc_path}/status') as f:
                        for line in f:
                            if line.startswith('Uid:'):
                                uid = int(line.split()[1])

                    # Command line.
                    with open(f'{proc_path}/cmdline') as f:
                        cmdline = f.read().replace('\0', ' ')

                # Process probably died.
                except:
                    pass
                
                # Print new process information with timestamp.
                time = datetime.now()
                time = time.strftime("%Y-%m-%d %H:%M:%S")
                print(f'{time} UID={uid}\tPID={proc_list[i]}\t| {cmdline}')

                # Linux assigns process IDs sequentially.
                pid = dir

if __name__ == '__main__':
    _main()

Now I’ll do the same demonstration from earlier and see if the script can still catch and print information about the cron jobs as they execute:

It looks like the script is printing information about each cron job as it executes, just like the last script. So far, it seems to function the same. Now I will check the system resource usage with the updated script executing on the left and top on the right:

As you can see, the system resource usage by the script has decreased dramatically. This would certainly look much less suspicious to anyone monitoring the system resource usage while the script is executing. Now that I’ve demonstrated how the inotify API can be used to check for new processes while not using a lot of system resources, I want to conclude this article by demonstrating why it’s bad practice to pass clear-text credentials on the command line in Linux.

Attack Demonstration

In this demonstration (thanks An00bRektn), I will assume I am an initial access broker (IAB) who has exploited a vulnerability in an exposed Linux web server on an enterprise company network. I now have remote code execution on the victim host as a low-privileged user (presumably www-data). At this point, I could go to my favorite hacking forum and sell my access to a ransomware operator. Wanting to make the most money as possible, I will first enumerate the host a bit more to check for anything interesting like exposed credentials or privilege escalation paths.

After some enumeration I find that another user named admin is logged into this host:

I want to know if they’re doing anything interesting. I retrieve my script from the Internet and run it, so that I may see any commands the user admin is executing:

The output of my script indicates the admin user has created a new Splunk Enterprise Docker container via the following command:

docker run -p 8000:8000 -e SPLUNK_PASSWORD=Iw5iwvGSshrZCzbfNAHb -e SPLUNK_START_ARGS=--accept-license -it --name so1 splunk/splunk:latest

• Notice that the command contains the clear-text password Iw5iwvGSshrZCzbfNAHb which is used to authenticate to the Splunk Enterprise instance.

It’s incredible that the user admin just happened to create a new Splunk Enterprise Docker container just after I started my script. What a coincidence! Okay, okay… obviously this situation was a bit of a stretch. A more likely scenario might be that an attacker gains access to the server, plants the script, lets it run indefinitely, and only forwards strings that look like credentials to another server. In any case, I hope this demonstration has made it clear to you why it’s bad practice to pass sensitive information on the command line in Linux.

By the way, why did that user include the password for the Splunk Enterprise instance in the command-line in the first place? Don’t they know this is bad practice? Surely the official Splunk Docker GitHub Repository doesn’t instruct people to include clear-text credentials in the command used to create new Splunk Enterprise Docker containers… right? Right?!?!

FAQ

Why not use the inotify API to monitor /proc for new subdirectories?

From proc(5):

The proc filesystem is a pseudo-filesystem which provides an interface to kernel data structures.

The directory /proc and its subdirectories aren’t actually files. It’s actually an interface that allows user space processes to communicate with kernel space.

I can’t read information about other processes under /proc!

The traditional behavior of everybody being able to access information in the /proc/[pid] directories can be changed by specifying a different mount option for the proc filesystem (see proc(5)).

The script in this article doesn’t catch the commands I execute.

In my experiments, the scripts I created for this article are very unpredictable and irregular. They seem to do a good job at catching cron jobs, but tend to miss many user-executed commands. I have a feeling this is because they are simply not fast enough at retrieving information from /proc. Furthermore, at the time of writing, the last commit to the PyInotify library repository was on August 26th of 2020. With that said, the scripts in this article are intended entirely for educational purposes. In practice, I highly recommend you use the popular pspy tool instead, which is actually what originally inspired me to write this article in the first place.