Over a million developers have joined DZone.

Read This Post, Especially if You Don’t Have Time

DZone's Guide to

Read This Post, Especially if You Don’t Have Time

If your systems are suffering from time skew, it could be effecting your security in multiple ways, such as throwing off your log correlation of security events.

· Security Zone ·
Free Resource

Discover how to provide active runtime protection for your web applications from known and unknown vulnerabilities including Remote Code Execution Attacks.

“Time keeps on slipin’ slipin’ slipin’, into the future” — “Fly Like An Eagle”, The Steve Miller Band

Your web server’s system time can keep on slippin’ far into the future if Network Time Protocol (NTP) is not properly configured. Having accurate system time is critical for application logic, scheduled jobs, and of course logging. With logging, if the system time is off, log forensics and log correlation of security events across systems become a nightmare, if not impossible. Many software components on your servers rely on accurate system time, including the Signal Sciences agent. The agent uses the local system clock to generate timestamps when logging detections and other events. As a result, ensuring accurate system time on all your web servers is a must for robust security — and this is especially true for virtual machine based deployments.

Why do I call out virtual machines? Our agent has been deployed in numerous environments and on thousands of machines (bare metal, virtual machines, containers, IaaS, PaaS, and whatever else you might imagine). In new deployments, virtual machines without NTP configured stick out like a sore thumb since their system time is way off. We’ve seen ranges from tens of thousands to hundreds of thousands of seconds off into the future! This drastic time skew has a severe impact on security related events since the timestamps can be so significantly skewed. One example of impact is when presenting events in a dashboard that is time-based, e.g. displaying all events within the last 6 hours. You won’t see the data if its timestamps are several hours in the future. Time skew can happen on any infrastructure, but if you are running virtualized infrastructure you’ll want to pay close attention.

In addition to NTP not being configured, there has been research published on attacks against the NTP protocol. Some of which result in shifting time on NTP clients. This research is published here and referenced in this Ars Technica article. Another threat consideration is a malicious insider, who could modify system time in attempts to hide events or manipulate time sensitive transactions.

“Watching” Time

How can we mitigate the risk of skewed timestamps, missing data, and possible attacks on system time? First, NTP is a beautiful thing. Make sure you have a consistent NTP implementation across all your servers. Second, periodically verify the system time on your servers is not skewed beyond an acceptable threshold. Fortunately for Signal Sciences customers, you already have visibility into your server time! Since time is such a crucial part of security, our agent provides visibility on your server clock skew. This is just one of the many metrics reported on by the agent, and it can be found via the UI by navigating to the agent’s graph page. On that page, scroll down towards the bottom to find the Agent Clock Skew graph.

The screenshot above is from a VM, where the clock skew is obviously drastically off until it was corrected.

The screen shot above is from an agent running in a popular PaaS environment. A one-second clock skew is relatively negligible, but depending on your circumstances perhaps it could be an issue.

 In the next section, I’ve published two scripts, one is a proof-of-concept Bash shell script that can be used to help monitor system time across many servers.

Monitoring Clock Skew

Ok, visibility on clock skew — great! Pretty graph — great! But what if you are like many of our customers and have hundreds of agents deployed. That’s a lot of looking at pretty graphs to find hosts with clock skew — not great! Good news, Signal Sciences makes all data available via its API — great!!! This means you now have the ability to automate monitoring of clock skew across your web server farm.

With the API you can integrate with any monitoring system you may have, however, I’ve provided an example of checking clock skew with just two scripts. The first script, SigSciApiPy, handles authenticating and pulling agent metrics from the API. The second script is shown below, clock_skew.py, loops through the agent data looking for the value of host.clock_skew. Host.clock_skew is the median number of seconds that the particular agent’s clock is skewed compared to the time on our servers, over the past 5 minutes. So for example, if we have “host.clock_skew”: 30, that means that we saw a median skew of 30 seconds over the past 5 mins. The clock_skew.py script will print a warning or alert message to the console based on defined thresholds.

#!/usr/bin/env python
# clock_skew.py
# This script reads json data from /tmp/sigsci-agents.json
# and prints a message if an agent's clock skew is greater
# or equal to the defined thresholds.
# SigSciApiPy (https://github.com/signalsciences/SigSciApiPy) is
# an easy way to pull agent data from the Signal Sciences API and
# save it to /tmp/sigsci-agents.json
# Example usage with SigSciApi.py:
# ./SigSci.py --agents > /tmp/sigsci-agents.json; ./clock_skew.py

# Thresholds
WARN  = 3

import json

agents = open('/tmp/sigsci-agents.json', 'r')
data   = json.load(agents)
skew   = 0
count  = 0


for agent in data['data']:
    skew = int(agent['host.clock_skew'])

    if skew >= ALERT:
        print("ALERT: Clock skew for {0} is {1}".format(agent['agent.name'], skew))
        count += 1

    elif skew >= WARN:
        print("WARNING: Clock skew on {0} is {1}".format(agent['agent.name'], skew))
        count += 1

if count == 0:
    print("Everything is running on time!")

Download or view the script on Github.

An example run of these scripts together looks like this:

./SigSci.py --agents > /tmp/sigsci-agents.json; ./clock_skew.py
WARNING: Clock skew on host1 is 4
ALERT: Clock skew for host2 is 5
WARNING: Clock skew on host3 is 4
WARNING: Clock skew on host4 is 3
ALERT: Clock skew for host5 is 5
ALERT: Clock skew for host6 is 6

Note, if you do use clock_skew.py, you’ll want to set the WARN and ALERT threshold variables to values that make sense to you.

Below is a script that can achieve a similar result in monitoring for clock skew. If shell scripts are not your thing, there are many other tools that may be a better fit in your environment. For example, monitoring systems like Nagios or Zabbix, or configuration management tools such as Ansible, Chef, or Puppet. I’m sure there are other, and even better, ways to monitor for clock skew. If you have suggestions please add them in the comments section.

#!/usr/bin/env bash
# clock_skew.sh - Check remote hosts for clock skew (time drift), and
# output a warning or alert message based on defined thresholds.
# Instructions:
# Create a hosts.txt file that contains the list of hosts to be checked.
# hosts.txt should contain one host entry per line, and username and
# port fields are optional. Format:
# <username>@<host or IP address>:<port>
# Note: this is a proof-of-concept script so prompting for a single
# password to access all the hosts in your host.txt file may not be
# feasible. Consider modifying the script to handle prompting for multiple
# passwords, or use ssh keys.
# Set threshold variables below to your tolerance level.

# Thresholds

# prompt for ssh password.
read -s -p "Enter password:" password
export SSHPASS=$password


# loop through hosts.txt file entries.
for host in `cat hosts.txt`;
IFS=':' read -a host_config <<< "${host}"


if [ ! -z ${host_config[1]} ];

# get local and remote date/time.
remote_time=`sshpass -e ssh -p ${p} ${h} date`

# convert the date/time to seconds from the Epoch.
local_sec=`date --date="${local_time}" +%s`
remote_sec=`date --date="${remote_time}" +%s`

# calculate the time skew.
skew=`expr ${remote_sec} - ${local_sec}`

# output messages if thresholds are met.
if [ $skew -ge $ALERT ];
echo "ALERT: Clock skew on ${h} is ${skew}."
elif [ $skew -ge $WARN ];
echo "WARNING: Clock skew on ${h} is ${skew}."

if [ $count -eq 0 ];
echo "Everything is running on time!"

Download or view the script on Github.

Monitoring for Time Skew Is a Security Must

I think we can all agree that ensuring you have accurate system time across your servers is critical for security. Also, having the ability to monitor for time skew is equally as important. I hope the example scripts I’ve provided help to jump start your clock skew monitoring efforts in a timely manner.

Find out how Waratek’s award-winning application security platform can improve the security of your new and legacy applications and platforms with no false positives, code changes or slowing your application.

security ,time skew ,loggging

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}