Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Detect Suspicious Linux Processes

DZone's Guide to

Detect Suspicious Linux Processes

Ever bothered by suspicious processes running on your servers? No doubt how dangerous they might be: valuable data leaked, CPU/memory wasted, or DDoS attack other victims, etc. Read this article to learn how to easily capture those annoying troublemakers.

· Performance Zone
Free Resource

Transform incident management with machine learning and analytics to help you maintain optimal performance and availability while keeping pace with the growing demands of digital business with this eBook, brought to you in partnership with BMC.

Ever bothered by suspicious processes running on your servers? No doubt how dangerous they might be: valuable data leaked, CPU/memory wasted, or DDoS attack other victims, etc. Want to learn how to easily capture those annoying troublemakers? Even better, learn how to get alerted without additional human effort.

List all non-kernel processes. Usually, kernel processes are safe and clean. For kernel processes, either PID(process id) is 2 or PPID(parent process id) is 2. Here is how to get all non-kernel processes.

# rss(resident set size): real RAM usage
# -deselect: rule out matched processes
root@denny:~# ps --ppid 2 -p 2 -p 1 \
   --deselect -o uid,pid,rss,%cpu,command
UID   PID   RSS %CPU COMMAND
   0   411  1848  0.0 /lib/systemd/systemd-
   0   572  2904  0.0 dhclient -1 -v -pf /r
 102   902  1244  0.0 dbus-daemon --system
   0   912  1948  0.0 /lib/systemd/systemd-
   0  5869   388  0.0 upstart-socket-bridge
 200  1953   904  0.0 /usr/sbin/apache2 -k
 200  3463  3700  0.0 /usr/sbin/apache2 -k
  ...  ...
  ...  ...
   0  5098  4224  0.0 sshd: ubuntu [priv]
   0  5139  1748  0.0 /usr/bin/python /usr/
 200  5140  3484  0.0 /usr/bin/python /usr/
 200  5176  1904  0.0 sshd: ubuntu@pts/3
 200  5177  3860  0.0 -bash
 200  5193  1200  0.0 tmux attach -t denny
   0  5297  4224  0.0 sshd: ubuntu [priv]
  ...  ...
  ...  ...

Rule out trusted processes. We may have many processes running, which are expected and trusted. e.g apache2, tomcat7, mysqld, etc. To avoid distraction, build a white list especially for your project.

Sort processes by memory and CPU. We're more concerned about suspicious processes using noticeable resources.

# Sort by memory first, then cpu
ps --ppid 2 -p 2 -p 1 --deselect \
  -o uid,pid,rss,%cpu,command, \
  --sort -rss,-cpu

Automate Detection Process and Get Alerts. We hide all the complexities and whitelist configuration in a python script (detect_suspicious_process.py). If you issue the python command, you may see output like "Identified processes count: XXX." Define a scheduled task to run periodical checks and confirm the number.

If the number is not 0 or it changes, send alerts. It might take a while to build a suitable white list. Once it's done, your servers are always more secured and managed!

wget -O /tmp/detect_suspicious_process.py \
https://raw.githubusercontent.com/\
DennyZhang/devops_public/tag_v2/python/\
detect_suspicious_process/\
detect_suspicious_process.py

# Detect suspicious process
python /tmp/detect_suspicious_process.py

# Detect by customized whitelist
python /tmp/detect_suspicious_process.py \
   --whitelist_file /tmp/whitelist.txt

Here is some more reading that you can do: Automate Insecure Ports Check By Nmap.

Evolve your approach to Application Performance Monitoring by adopting five best practices that are outlined and explored in this e-book, brought to you in partnership with BMC.

Topics:
linux ,devops ,security

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}