Logging on Linux: What to Look Out For With Lots of Linux Boxes
The DevOps Zone is brought to you in partnership with LogEntries. Learn how you can convert any query into a data visualization for your dashboard with Logentries.
While people generally understand that it is important to store and manage logs so that they can be regularly reviewed for operations, root cause analysis, security purposes etc… Oftentimes we find users struggle with figuring out where to start when reviewing their log data, and what they should be looking out for. A common question we get is:
“ok so I’ve a bunch of linux boxes, I’m running a LAMP stack, what should I be looking out for in my logs”.
Unfortunately, this is a little like trying to explain to someone how to fix their car if it is having problems, WITHOUT knowing what the problem is. So it can be very difficult to say: “well you should be looking in log X for problem Y”. We find a much better answer is to give people an overview of logging on their platform, how it works, how it can be configured, what files & locations different event types are logged to and what tools can be used to analyse the data. So that’s exactly what we are going to do here. That way if you do want to perform a security review, or need to ‘take out’ some bugs by digging into your log data, you will know where to look and will have the right tools for the job.
We’ll be covering a number of platforms and technologies over the coming weeks and months, but as a starting point we’ll begin with an overview of logging on linux and what you need to know if you are running a LAMP stack in particular.
All linux (and UNIX) distributions as well as most routers, firewalls and network devices use Syslog to provide logging functionality. Syslog is often referred to as the “heterogenous network logging workhorse”…. ok so maybe not that often…. but the phrase does sum up pretty well what syslog is.
To be precise Syslog is actually a protocol which allows a machine/device to send log data across the network to event message collectors (syslog servers). The protocol defines a messaging architecture, what transport layer protocols and ports should be used for sending information, how messages should be constructed, as well as a way to describe where the messages have come from in your system (facility levels) and how they should be treated (severity levels).
The term ‘Syslog’ however is also regularly used to refer to any of the protocol implementations that ship with the different operating systems, such as syslogd, rsyslog, and syslog-ng. They all support collection and forwarding of logs to a log server or logging service. The different implementations differ in terms of the functionality they provide with some being more powerful than others. Syslogd is the simplest implementation, and supports UDP forwarding only. Rsyslog and syslog-ng provide some extra and quite useful functionality. Both support UDP and TCP forwarding for example. Rsyslog ships with Ubuntu and Fedora, and from my own experience in toying with it, it is dead simple to configure. As well as handling your OS level logs it provides support for common components like MySql, PostgreSQL, Oracle and others. Syslog-ng, which is maintained by the guys at Balabit, provides some extra features like the ability to classify, tag, and correlate log messages in real-time. It can also be configured to forward non-syslog logs (e.g. produced by your application servers/application code) to third party services (like Logentries) or to Syslog collection servers. If you are a windows user and are feeling a bit left out with all this talk of syslog and linux, do not despair there’s even a syslog implementation of sorts for windows (snare)!!
To configure Syslog you need to deal with the relevant ‘.conf’ file for your syslog version. With rsyslog that’s rsyslog.conf and usually located at ‘/etc/rsyslog.conf’. The config file is where you specify things like:
- the format of your log messages.
- what file events from the different facilites (i.e. different parts of your system) should be logged to.
- what file events with different severities should be logged to.
- where to log your data – e.g. you can log to files, but you can also log to databases or forward your logs to a remote server/logging service.
- filters – i.e. remove events that you do not want logged.
Note if you’re playing with syslog and you want to create test log messages the ‘logger’ command can be very useful. The ‘logger’ command allows you to send some test log events to syslog. For example:
$ logger this is a test log event
Will send the message ‘this is a test log event’ to the file ‘/var/log/syslog’ with the default rsyslog configuration. ‘logger’ can also be used to add logging functionality to any shell scripts you write.
2. Where Linux Logs Live
If you are on a linux distro, most of your logs are going to live in the ‘/var/log folder’ or in one of its sub folders. The Geek Stuff give a list the 20 different log files that are located under ‘/var/log/’ directory which can be of interest and give details about what events are generally contained within these logs.
Here’s a brief summary of some of the more interesting of these, note some of these are distribution specific:
- /var/log/message: Global system messages, including messages logged during system startup
- /var/log/dmesg: Kernel ring buffer information, contains information on hardware devices that the kernel detects during boot process. You can use the ‘dmesg’ command to list these events from the command line.
- /var/log/auth.log: Authenication logs, includes user logins and the authentication mechanism used.
- /var/log/daemon.log: The log file for any processes running in the background.
- /var/log/kern.log: Kernel logs, all messages output by the linux kernel.
- /var/log/cron.log: Scheduled cron job logs.
- /var/log/maillog: Mail server logs.
- /var/log/boot.log: Messages generated during the boot sequence.
- /var/log/secure: Authentication log
Also if you are running a LAMP stack these may be of interest:
- Apache: apache web server logs (i.e. access.log and error.log) are usually found at /varl/log/httpd or /var/log/apache2/
- Nginx: if you run nginx rather than apache, nginx logs are located
at /var/log/nginx/. Nginx log events are stored in the
yourapp.access.log and yourapp.error.log. The access log contains
records of requests made to the web server, with any related errors
being sent to the error.log. Further info on nginx logs can be found at nginx.org.
- MySQL logs: The general query log, error log and slow query log are
usually of interest and are located at /var/log/mysql.log,
/var/log/mysql.err and /var/log/mysql/mysql-slow.log respectively. Note
the slow query log often needs to be enabled in the relevant mysql
We’ll be covering technologies such as .NET, Java, Ruby, PHP etc. in more details in future posts, giving details on what logs are important for these technologies and what to look out for within the log data.
3. Things to look out for
“What to look out for in linux logs” was the motivation for this
post. And although I’ve said above that this is a difficult question to
answer without knowing the specifics I thought I’d better not dodge the
question completely and at least give some examples of issues and how
they can be investigated. Sandeep Kalathil gives some nice examples in
his piece on the Binbert blog and so also do the guys at Linucium. I’ll cover some of these here and give a few more also:
- messages.log: is known as the “general system activity” log and is
where various system applications and daemons record messages as well as
messages from non-kernel boot issues, cron jobs, authentication events,
and messages that go to the dmesg log end up. It is usually the first
place to look when there is something up and you need to begin
- dmesg.log: As explained above, this log contains messages from the
kernel that appears during the boot process, thus it is the first place
to look if your system is not behaving properly and you suspect
something went wrong during start up, such as a device driver issue,
wifi issue or a service that fails to start correctly. For example if
you suspect your wifi driver has problems, you might search the
dmesg.log file for the string ‘wlan’ or ‘wifi’ like this guy.
Checking my own dmesg.log (I’m running ubuntu) I see that my MongoDB
instance failed to start correctly. I get the following when searching
for ‘init’ in dmesg.log:
18.480052] init: mongodb main process (1273) terminated with status 100)
Copying the error code into google gets me a quick fix also – nice! Finally the guys at Ghacks also give some good tips on identifying issues, related to attaching usb devices, using the dmesg log.
- Logins/dictionary attacks: If you suspect that there is a security
issue with system access you may want to go directly to the ‘auth.log’.
The ‘auth.log’ records all user logins and the login mechanism used, and
will capture any unauthorised access or attempted access for that
matter. For those with servers that have publicly facing IPs this is a
log that you should keep an eye on, in particular to monitor dictionary
attacks which WILL occur regularly enough. My recent post on the Engine Yard blog shows what a successful dictionary attack looks like. It’s a good idea to create alerts around this type of activity in your logs also.
- Tracking sudo access: whenever you run the ‘sudo’ command, it logs
it in the auth.log file in case you need a clear picture of who did
what, and when
- kernel.log: Diagnosing problems with a new kernel installation (-e.g. distribution upgrade) or major system problems
- Apache logs: apache logs
can be reviewed for issues with web requests such as broken links
(check the apache error log for ‘”file not found’ or ’404′) or for
security issues such as denial of service (DOS) attacks. The apache logs
contain the IP address of the machine that made the request, so you can
see who is trying to accessing your web application. If someone is
performing as DOS attack you can, as a starting point, figure out where
this is coming from and if it is a DOS or not. Some apache common error
codes are discussed in this post. The full list of possible status codes can be found in the HTTP specification (RFC2616 section 10).
- MySQL logs: Database logs can be useful for investigating slow queries
or DB related errors. Slow queries are essentially queries in your
database that are taking too long. They can result in slow running
applications and are one of the most common issues resulting in
application performance problems.
I hope this helps in giving you a clearer picture of logging on linux and where to start if you need to review your logs. In part 2 of our ‘logging on linux’ series, we’ll cover a number of other topics such as tools for analaysing and managing your linux logs, log forwarding on linux and why storing logs remotely is recommended.