Navigating Through Logs for Information Disclosure Requests

DZone 's Guide to

Navigating Through Logs for Information Disclosure Requests

In this article, we discuss how to navigate through logs for information disclosure requests with Spectx.

· Big Data Zone ·
Free Resource

In a world of compliance and disclosure requests, the ability to investigate raw log files whilst shutting out the noise can not only be a time-saving maneuverer in your process but also reduce the risk of mistakes. The ability to analyse large volumes of log files, be it on the cloud, or hidden away in on-prem archives, will make a great difference on how your tech team operates.

Using higher education as an example. Every year, new students join a University and for IT teams, this means new logs. But it also means new devices on the networks, in Europe, this includes Eduroam, a 3rd party network point where logs may not be as easily accessible. On average, a student will bring in a mobile phone & laptop. But in this ever-growing IoT world, students are expected to bring more smart devices as well as devices such as tablets. This increases a student’s footprint on any SIEM solution.

A problem often faced with universities is that every department is run differently (and often work in silos). Systems running under a lecturer’s desk on a machine that was decommissioned 4 audits ago are a common find. This can make compliance audits difficult, but when it now comes to data disclosure requests by authorities (Department for Work and Pensions, local councils, HMRC, Police and Visas and Immigration), the trouble begins. 

When partnered with the red tape around disclosure requests from the police, this can often be time-critical so by the time the request has gone through the procedure and policies, there is often now a battle against time for the IT department to work on producing. And for most universities and even other local government organizations, they do not have 24/7 SOC environments where they can dedicate resources to these requests. With pricing models of SIEMs often meaning certain systems are not included in the monitoring, having a tool that can tackle this challenge is key. 

For illustrating this post, I will use SpectX Desktop — a free edition of a log forensics tool that can query raw log files and blobs directly from their storage on-prem and in the cloud, skipping ingestion, and indexing.  With legacy Linux systems, for example, an analyst will spend more time working on the perfect grepping script than with the actual data and then again trying to format the data to make it readable and understandable. 

For local governments and higher education establishments, in this current financial climate, ensuring resources are utilized is key to growth and sustainability. This includes staff on the IT team being able to utilize their time effectively. But, as data is often in different data sources and depending on the SIEM, only a limited volume of data is ingested, indexed and visible; a tool such as SpectX can quickly marry the data together from different sources.

For example, a request universities often get relates to did student X attend this University during a Y period of time. A simple check of academic records will confirm that. But then it becomes more complex and requires querying multiple data sources, “when X students attended your institute, do you have logs confirming the activities they carried out online. We want to see if they browsed www.badstuff.org/NotARealSite/, and we think they are involved in a malicious campaign of phishing which originated from a University IP.” 

As this request filters down departments and procedures, for the analyst whose lap it will land on, being able to accurately combine data, brush out noise, and query the data in a logical manner will be key. This request comes with data dumps of the day requested from various systems. This scenario plays out across different IT teams in many institutes.

So let’s look at how simple it is to analyze logs:

Log output

An analyst will receive dumps of logs covering a particular date and be seeking either a certain date or a user and being able to query. Producing a report in a readable format without having to brush up your grep skills will also help during a time-critical stage. So to see if a user has been on a system during a certain time period, we can connect the system logs to SpectX and then parse and search through them. 

Limiting the search to a certain time period, going through multiple raw log files and blobs stored locally, as well as in S3 with SpectX is as easy as:


The result from the query we are left with is that the user ‘john’ did access the system at that timestamp from that geolocation.

Log output


Ease of use, especially when it comes to dealing with logs, makes the process less tedious for analysts (with SpectX it becomes fun, check out the post on Analysing Git Logs). Being able to experiment and try different queries, views, and commands directly on raw logs whilst working on a request will not only increase the accuracy of investigations, but it will also add a new dynamic to dealing with these requests. For more information about SpectX, see the documentation.

forensics ,incident response ,law enforcement ,log aggregation ,log analytics ,log data streams ,log file ,log files ,security analytics ,siem

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}