DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
  1. DZone
  2. Data Engineering
  3. Data
  4. Capturing Request Logs From Signal Sciences

Capturing Request Logs From Signal Sciences

This post looks at capturing request data logs from Signal Sciences in cybersecurity monitoring.

Phillip Maddux user avatar by
Phillip Maddux
·
Aug. 17, 18 · Tutorial
Like (1)
Save
Tweet
Share
3.41K Views

Join the DZone community and get the full member experience.

Join For Free

Logs are important. We need them to investigate, monitor, and analyze. In cybersecurity, we have many tools for generating logs, and extracting those logs to a central repository is a common routine. However, the methods and tools for capturing logs are not always common. In this post, I'll provide an overview of several options available to users of Signal Sciences for capturing request logs.

With Signal Sciences, there are a few types of log data. At a high level, these are request data and diagnostic data. Request data contains information about requests hitting the site(s) you are protecting with Signal Sciences. This is the data you want for threat monitoring and analysis. Diagnostic data contains information about Signal Sciences modules and agents This is the data you want if you need to perform advanced monitoring over the health of the various Signal Sciences components you've deployed.

This post will focus on request data — I may cover capturing diagnostic data in a future post. In addition, this post is meant to be a high-level introduction to several available options, not a comprehensive guide.

Request Logs

The primary option for extracting request data from Signal Sciences is the request feed REST API endpoint, which is documented here, and there is a companion document with details on extracting your data here. I highly recommend you read the document on extracting your data as it explains some of the restrictions with the API endpoint.

Now that you are familiar with the API endpoint, you can always script your own solution in any language to download the data. However, if scripting is not your thing, here are some options on how you can download the data.

SigSciApiPy Python Script

The SigSciApiPy script is meant to be a sample reference script for calling the API, but it is very functional and handy. The following command can be used to pull the last 24 hours of request feed data:

 ./SigSci.py --feed --from=-24h 

Note the default output format is JSON. You can change the format to CSV format with the "format" command-line option, but you will notice some of the request metadata will be missing. The following is an example:

 ./SigSci.py --feed --from=-24h --format csv 


Splunk App

If you have Splunk then you'll likely want to use the Splunk TA (technology add-on). This TA calls the API to download data and can be configured to download data for multiple sites.

Logstash (or the Elastic Stack) Plugin

If you have Logstash, you know it can pump logs out to any destination via a variety of output plugins. Typically, that destination is Elasticsearch. However, before you can pump out logs, you need to first get them via an input plugin. There is now a Logstash input plugin for Signal Sciences available that can get you up and running quickly.

Syslog

REST APIs not your thing? Maybe your logging tools are centered around ingesting data in Syslog format. In addition, you may want to capture the data directly from Signal Sciences agents. There are a few options for you.

Tail the Log

The first option involves tailing the agent's log file. Configure the agent to log to a file with the following (default location for the agent configuration file on Linux: /etc/sigsci/agent.conf):

debug-log-web-inputs = 1
debug-log-web-outputs = 1
log-out = "/var/log/sigsci.log"

Since Syslog wants a single flat line per record, we are setting debug-log-web-inputs and outputs to 1. This will produce a flat single line per record, rather than the default multi-line JSON format.

Next, start the agent and then run the following command as a backgrounded process to forward the logs to the systems Syslog:

 tail -F /var/log/sigsci.log | logger &

When using this method, you should also consider rotating the log file and removing older files to preserve disk space. Using the logrotate tool can easily help with managing the log files.

sigsci-syslog-client

An alternative to using the logger and logrotate commands mentioned above is the sigsci-syslog-client utility. This tool is written in Go and helps to easily forward the logs to a local or remote Syslog server. It also has a companion script to manage the max file size for the local log file to preserve disk space.

Conclusion

Having the ability to extract request data and easily import it into your security information and event management (SIEM) system or other log analysis tools is a must. Whether you prefer JSON or Syslog format, the options outlined above will help you get started with capturing Signal Sciences request logs. If you are a Signal Sciences customer and have questions on capturing log data, please don't hesitate to contact the support team or your Technical Account Manager.

Requests Signal Data (computing)

Published at DZone with permission of Phillip Maddux, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • The Path From APIs to Containers
  • How To Build a Spring Boot GraalVM Image
  • 5 Steps for Getting Started in Deep Learning
  • Best CI/CD Tools for DevOps: A Review of the Top 10

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: