Centralized Linux Bash History

Most of the time, a user interacts with the Linux system through shell commands. The default installation of Linux provides some level of storing this information. But these are standalone and cannot predict the anomalies associated with not having session correlation in it. Additionally, a common user can override and hide his own activities.

Approach 1

By default, Linux records all the commands executed in a .bash_history file.

Advantages

• You can’t remove the file
• You can’t change the file permissions

[user1@rhel-host1 ~]$ chmod 700 .bash_history
chmod: changing permissions of ‘.bash_history’: Operation not permitted
[user1@rhel-host1 ~]$ rm .bash_history
rm: cannot remove ‘.bash_history’: Operation not permitted
[user1@rhel-host1 ~]$

Drawbacks

• Not centralized; each user gets their own .bash_history file.
• There is no user session correlation.
• The user can override the default behavior and commands will not be saved.

[user1@rhel-host1 ~]$ export HISTCONTROL=ignorespace ;
[user1@rhel-host1 ~]$ <SPACE>  ls -ltr  ##  add <SPACE in front of the command >  

Approach 2

 At a regular interval, ship the Bash history to a centralized server and look through all the commands.

Advantages

• Better than the default and there is no need to look at multiple files

Drawbacks

• You need an agent like Filebeat or Logstash to forward the commands to a centralized system.
• Still provides no user session correlation

Approach 3

Use the user command prompt variable (PROMPT_COMMAND) to send the last executed command to a centralized system.  

Advantages

• No agent is required

Drawbacks

• A user can reset the variable; this will eventually stop sending the commands to the centralized system

Approach 4

 Enable the trap command and capture the user command to a centralized system. 

Advantages

• No Agent required

Drawbacks

• A user can disable the trap.

Preferred Approach 

Building Blocks

trap-trap (Enhanced): 

Enable the trap  command at the user session start time (/etc/bashrc) and start logging the commands to our centralized platform.

Profile functions are enhanced to check user commands. if they try to override the expected behavior, the script will overwrite the command. Also, auto-heals /re-enable the trap to some extent.

It traps the remotely executed command and adds a tag to differentiate the regular commands. Also injects user’s session details to the logger.

More information on script documentation and the Docker file of building blocks can be found here.

function log2syslog
{
## set the Hostname of present system (Client)
   HOSTNAME=$(hostname) 

## Set the Timestamp of command Execution with Date RFC-5424
   CMD_EXEC_TIME=$(date '+%Y-%m-%dT%H:%M:%S%:z') 

## set exec the Date
   CMD_EXEC_DATE=$(date '+%Y-%m-%d')

## Gets the Time
   CMD_EXEC_TIME_ALONE=$(date '+%H:%M')

## Checks for user’s origin IP is Empty and set equivalent mock data 

   [[ -n ${SSH_CLIENT} ]] || SSH_CLIENT="noData noData noData"  

   declare WHO_RUNS 
## Variable declared for getting logged-in user details

   WHO_RUNS=$(who -m | sed 's/[()]//g' | sed -r 's/ +/ /g')
  ## Gets the Logged-in User ID, IP & Login Time

   [[ -n  ${WHO_RUNS} ]] || WHO_RUNS="noData noData ${CMD_EXEC_DATE} ${CMD_EXEC_TIME_ALONE} noData"  
 ## Checks for user’s origin details are Empty and setting string equivalent mock data 

   declare COMMAND
## Variable declared for getting logged-in user executed Command

   COMMAND=$(fc -ln -0 | sed -r 's/     //g' | sed -r 's/ +/ /g') 
## Fetch the executed Command

[[ -z ${BASH_EXECUTION_STRING} ]] || COMMAND="#000 RemotelyExecCommand ${BASH_EXECUTION_STRING}"
## Checks for Remotely (via ssh) Executed Commands and  set the  COMMAND variable 

   [[ -n ${COMMAND} ]] || COMMAND="#000 CommandIsEmpty_or_ssh_login"
## Checking for executed Command and setting string equivalent mock data if empty


## Checks if user has tried to Disable the Trap; then  SigQuit the PID ; this will reenable the logger trap

   if [[ ${PREVIOUS_COMMAND} != *" CommandIsEmpty_or_ssh_login" && -n  ${PREVIOUS_COMMAND} &&  ${COMMAND} == *"trap"*"-"*"DEBUG"* ]]; 

   then
        kill -SIGQUIT $$                   
    fi

## Checks whether HISCONTROL variable was modified; then reset it
  if [[ ${PREVIOUS_COMMAND} == *"HISTCONTROL"*"="* ]]; 
   then
    export HISTCONTROL="ignoredups"  
  fi

## Checks if User has tried to remove the Trap; then enable it back
 if [[ ${PREVIOUS_COMMAND} == *"trap"*"-"*"SIGQUIT"* ]];
   then
    trap enabletrap SIGQUIT
  fi

## Check for deduplicate and remove for redundant commands being logged; also hard-code the hostname in the logger

   if [[ -z ${PREVIOUS_COMMAND} || "${COMMAND}" != "${PREVIOUS_COMMAND}" ]] ;
    then
      logger -n xyz.abc.com -t cmdhist -i -- "${HOSTNAME} ${CMD_EXEC_TIME} ${SSH_CLIENT} ${WHO_RUNS} ${USER} ${COMMAND}"   
      export PREVIOUS_COMMAND=${COMMAND} 
   fi
}

## enable the trap in Debug mode
trap log2syslog DEBUG 

function enabletrap {
        trap log2syslog DEBUG  
}

## Enabling the Trap
trap enabletrap SIGQUIT  

Key Features

Benefits