Linux Machine Compromised: Power of Observation

Introduction

In debugging any issue or any dealing with any problem or circumstance, 2 things are important:

• Observation — Observation not only at the time of issue but in general times also.
• Combining your general observations and observations at the time of issue to conclude something.

In this blog, I will explain the following :

• What was happening on my machine?
• How I came to know my machine is broken into — Power of observation.

What Was Happening on My Machine

• The load on my machine is going very high.
• On top command, one process ./kwsapd0 is consuming around 3000% CPU.

From here we get to know that kswapd is a consuming process. The process kswapd0 is the process that manages virtual memory. So I thought that may be our some process is consuming more RAM and Virtual Memory is being used due to which kswapd process is doing its work but after hours of debugging, we found no process is consuming RAM and around 80% of RAM was free.

How I Came to Know My Machine Was Broken Into — Power of Observation

There were two general observation which I observed and helped my getting to know what was the issue:

• First, the kswapd process in the top command looks like [kswapd] not ./kswapd.
• Kswapd0 can only consume 100% as it uses only one core in the machine.

From there I got to know that this kswapd0 was something unusual. On further debugging I found ./.configrc/a/kswapd0 in root users directory.

Contents of this directory were:

$ find .configrc -type f
.configrc/dir2.dir
.configrc/a/kswapd0
.configrc/a/dir.dir
.configrc/a/a
.configrc/a/bash.pid
.configrc/a/run
.configrc/a/stop
.configrc/a/init0
.configrc/a/.procs
.configrc/a/upd
.configrc/cron.d
.configrc/b/sync
.configrc/b/dir.dir
.configrc/b/a
.configrc/b/run
.configrc/b/stop

There was also an entry in cron to run this.

So, from all of this, I learned that my system was compromised.

Yet I was unable to find out how my system was broken into. But in my future blog, I will explain what things one can check if your system is compromised and how it is compromised and what all security we can apply to our system to make it less hackable.