Say you have issued a command in your servers. Typically, the command might either back up something or perform a critical hotfix.

Surely, you know the start time of the process. But when it will end? How can you find the execution time when the process has already been started?

Getting the start time of a running process is easy. Either use the ps CLI or query the /proc file system directly.

Via ps command line: 

# Show process start time by pid
ps -o lstart= -p $pid

# Show process start time, cmd by pid.
ps -o lstart,cmd $pid

Via the /proc file system directly:

awk '{print "CPU time: " $14+$15; print "start time: " $22}' \
  /proc/$pid/stat

Getting the process execution time is  if we can control how it is started. Simply use time $command.

Wrap the command by adding time as a prefix. Super easy. Isn’t it?

time cp /etc/hosts /tmp/ 
# real 0m0.004s 
# user 0m0.000s 
# sys 0m0.000s 

But what if you have forgotten to add the time prefix somehow and you have no estimation when it will finish? How can you still get the execution time while the process is already running?

Well, you can certainly keep watching the screen and figure it out sooner or later. But a pro wouldn’t do that. That's pretty boring and time-wasting.

We all know the watch command, right?

The tee command might be less well-known. It reads from standard input and writes to both standard output and files.

# Record process start time to a log file
ps -o lstart= -p $pid > /tmp/watch_process_$pid.log

# Check process every 2 seconds, using watch command.
# When process is running, record current time to log file
watch "ps -o lstart= -p $pid && \
  (date | tee -a /tmp/watch_process_$pid.log)"

From the log file, we can get:

• The process starting time with the first entry of the log file.
• The ending time of the last entry of the log file.

Why? If the process has finished, the ps command will fail, and the following clauses will be skipped. So, with this method, we can monitor any existing process. Sounds great, isn’t it?

Any further improvements?

You can get a timely notification when the process ends. The process could have been done for several hours before you even remember to log in and check the status via the log file.

Let’s say that you have wrapped up send_alert.sh. This script will send alerts properly. Make a slight change, you will get alerts properly.

# Record process start time to a log file
ps -o lstart= -p $pid > /tmp/watch_process_$pid.log

# Check process every 2 seconds, using watch command.
# When process is running, record current time to log file.
# When process has finished, send_alert.sh will be executed.
watch "(ps -o lstart= -p $pid || (send_alert.sh && false)) \
  && date | tee -a /tmp/watch_process_$pid.log"

Then we’re all set!