Netstat and logging

Certify and Increase Opportunity.
Be
Govt. Certified Linux Administrator

Netstat

netstat is a useful tool for checking your network configuration and activity. It is in fact a collection of several tools lumped together. netstat or network statistics, is a command-line tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface (network interface controller or software-defined network interface) and network protocol statistics.

When you invoke netstat with the –r flag, it displays the kernel routing table in the way we’ve been doing with route. On vstout, it produces:

# netstat -nr

Kernel IP routing table

Destination   Gateway     Genmask         Flags MSS Window irtt Iface

127.0.0.1     *           255.255.255.255 UH       0 0         0 lo

172.16.1.0   *           255.255.255.0   U       0 0         0 eth0

172.16.2.0   172.16.1.1   255.255.255.0   UG       0 0         0 eth0

The –n option makes netstat print addresses as dotted quad IP numbers rather than the symbolic host and network names. This option is especially useful when you want to avoid address lookups over the network (e.g., to a DNS or NIS server).

The second column of netstat ’s output shows the gateway to which the routing entry points. If no gateway is used, an asterisk is printed instead. The third column shows the “generality” of the route, i.e., the network mask for this route. When given an IP address to find a suitable route for, the kernel steps through each of the routing table entries, taking the bitwise AND of the address and the genmask before comparing it to the target of the route.

The fourth column displays the following flags that describe the route:

• G The route uses a gateway.
• U The interface to be used is up.
• H Only a single host can be reached through the route. For example, this is the case for the loopback entry 127.0.0.1.
• D This route is dynamically created. It is set if the table entry has been generated by a routing daemon like gated or by an ICMP redirect message
• M This route is set if the table entry was modified by an ICMP redirect message.
• ! The route is a reject route and datagrams will be dropped.

The next three columns show the MSS, Window and irtt that will be applied to TCP connections established via this route. The MSS is the Maximum Segment Size and is the size of the largest datagram the kernel will construct for transmission via this route. The Window is the maximum amount of data the system will accept in a single burst from a remote host. The acronym irtt stands for “initial round trip time.” The TCP protocol ensures that data is reliably delivered between hosts by retransmitting a datagram if it has been lost. The TCP protocol keeps a running count of how long it takes for a datagram to be delivered to the remote end, and an acknowledgement to be received so that it knows how long to wait before assuming a datagram needs to retransmitted; this process is called the round-trip time. The initial round-trip time is the value that the TCP protocol will use when a connection is first established. For most network types, the default value is okay, but for some slow networks, notably certain types of amateur packet radio networks, the time is too short and causes unnecessary retransmission. The irtt value can be set using the route command. Values of zero in these fields mean that the default is being used.

Finally, the last field displays the network interface that this route will use.

Parameters – Parameters used with this command must be prefixed with a hyphen (-) rather than a slash (/). If a parameter is supported only on some platform or platforms, the platform or platforms is listed in parentheses after the parameter.

Parameter	Details
-a	Displays all active connections and the TCP and UDP ports on which the computer is listening.
-e	Displays ethernet statistics, such as the number of bytes and packets sent and received. This parameter can be combined with -s.
-g	Displays multicast group membership information for both IPv4 and IPv6
-i	Displays network interfaces and their statistics
-m	Displays the memory statistics for the networking code (STREAMS statistics on Solaris).
-n	Displays active TCP connections, however, addresses and port numbers are expressed numerically and no attempt is made to determine names.
-p (Linux)	Show which processes are using which sockets (you must be root to do this)
-r	Displays the contents of the IP routing table.
-s	Displays statistics by protocol. By default, statistics are shown for the TCP, UDP, ICMP, and IP protocols. If the IPv6 protocol for Windows XP is installed, statistics are shown for the TCP over IPv6, UDP over IPv6, ICMPv6, and IPv6 protocols. The -p parameter can be used to specify a set of protocols.
-t (Linux)	Display only TCP connections.
Interval	Redisplays the selected information every Interval seconds. Press CTRL+C to stop the redisplay. If this parameter is omitted, netstat prints the selected information only once.
-h (unix)	Displays help at the command prompt.

Netstat provides statistics for the following:

• Proto – The name of the protocol (TCP or UDP).
• Local Address – The IP address of the local computer and the port number being used. The name of the local computer that corresponds to the IP address and the name of the port is shown unless the -n parameter is specified. An asterisk (*) is shown for the host if the server is listening on all interfaces. If the port is not yet established, the port number is shown as an asterisk.
• Foreign Address – The IP address and port number of the remote computer to which the socket is connected. The names that corresponds to the IP address and the port are shown unless the -n parameter is specified. If the port is not yet established, the port number is shown as an asterisk (*).
• State – Indicates the state of a TCP connection. The possible states are as follows: CLOSE_WAIT, CLOSED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, LAST_ACK, LISTEN, SYN_RECEIVED, SYN_SEND, and TIME_WAIT.

Logging

Log files are files that contain messages about the system, including the kernel, services, and applications running on it. There are different log files for different information. For example, there is a default system log file, a log file just for security messages, and a log file for cron tasks. Log files can be very useful when trying to troubleshoot a problem with the system such as trying to load a kernel driver or when looking for unauthorized login attempts to the system.

Some log files are controlled by a daemon called rsyslogd. The rsyslogd daemon is an enhanced replacement for previous sysklogd, and provides extended filtering, encryption protected relaying of messages, various configuration options, input and output modules, support for transportation via the TCP or UDP protocols. Note that rsyslog is compatible with sysklogd.

Almost all logfiles are located under /var/log directory and its sub-directories on Linux. You can change to this directory using the cd command. You need be the root user to view or access log files on Linux or Unix like operating systems. You can use the commands to see the log files, as – less, more, cat, grep, tail, zcat, zgrep and zmore.

Common Linux log files names and usage

• /var/log/messages : General message and system related stuff
• /var/log/auth.log : Authenication logs
• /var/log/kern.log : Kernel logs
• /var/log/cron.log : Crond logs (cron job)
• /var/log/maillog : Mail server logs
• /var/log/qmail/ : Qmail log directory (more files inside this directory)
• /var/log/httpd/ : Apache access and error logs directory
• /var/log/lighttpd/ : Lighttpd access and error logs directory
• /var/log/boot.log : System boot log
• /var/log/mysqld.log : MySQL database server log file
• /var/log/secure or /var/log/auth.log : Authentication log
• /var/log/utmp or /var/log/wtmp : Login records file
• /var/log/yum.log : Yum command log file.

Various logs needed for system administration are

• Authorization Log – The Authorization Log tracks usage of authorization systems, the mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd and so on. The Authorization Log file may be accessed at /var/log/auth.log. This log is useful for learning about user logins and usage of the sudo command. Use grep to cut down on the volume. For example, to see only information in the Authorization Log pertaining to sshd logins, as – grep sshd /var/log/auth.log | less
• Daemon Log – A daemon is a program that runs in the background, generally without human intervention, performing some operation important to the proper running of your system. The daemon log at /var/log/daemon.log and contains information about running system and application daemons such as the Gnome Display Manager daemon gdm, the Bluetooth HCI daemon hcid, or the MySQL database daemon mysqld. This can help you trouble-shoot problems with a particular daemon.
• Debug Log – The debug log at /var/log/debug and provides detailed debug messages from the Ubuntu system and applications which log to syslogd at the DEBUG level.
• Kernel Log – The kernel log at /var/log/kern.log provides a detailed log of messages from the Ubuntu Linux kernel. These messages may prove useful for trouble-shooting a new or custom-built kernel, for example.
• System Log – The system log typically contains the greatest deal of information by default about your Ubuntu system. It is located at /var/log/syslog, and may contain information other logs do not. Consult the System Log when you can’t locate the desired log information in another log. It also contains everything that used to be in /var/log/messages.
• Application Logs – Many applications also create logs in /var/log. If you list the contents of your /var/log subdirectory, you will see familiar names, such as /var/log/apache2 representing the logs for the Apache 2 web server, or /var/log/samba, which contains the logs for the Samba server.

Syslog – All linux (and UNIX) distributions as well as most routers, firewalls and network devices use Syslog to provide logging functionality. To be precise Syslog is actually a protocol which allows a machine/device to send log data across the network to event message collectors (syslog servers). The protocol defines a messaging architecture, what transport layer protocols and ports should be used for sending information, how messages should be constructed, as well as a way to describe where the messages have come from in your system (facility levels) and how they should be treated (severity levels).

The term ‘Syslog’ however is also regularly used to refer to any of the protocol implementations that ship with the different operating systems, such as syslogd, rsyslog, and syslog-ng. They all support collection and forwarding of logs to a log server or logging service. The different implementations differ in terms of the functionality they provide with some being more powerful than others. Syslogd is the simplest implementation, and supports UDP forwarding only. Rsyslog and syslog-ng provide some extra and quite useful functionality. Both support UDP and TCP forwarding for example.

To configure Syslog you need to deal with the relevant ‘.conf’ file for your syslog version. With rsyslog that’s rsyslog.conf and usually located at ‘/etc/rsyslog.conf’. The config file is where you specify things like:

• the format of your log messages.
• what file events from the different facilites (i.e. different parts of your system) should be logged to.
• what file events with different severities should be logged to.
• where to log your data – e.g. you can log to files, but you can also log to databases or forward your logs to a remote server/logging service.
• filters – i.e. remove events that you do not want logged.

To create test log messages the ‘logger’ command can be very useful. The ‘logger’ command allows you to send some test log events to syslog. For example:

$ logger this is a test log event

Will send the message ‘this is a test log event’ to the file ‘/var/log/syslog’ with the default rsyslog configuration. ‘logger’ can also be used to add logging functionality to any shell scripts you write.

Important logs to monitor

• log: is known as the “general system activity” log and is where various system applications and daemons record messages as well as messages from non-kernel boot issues, cron jobs, authentication events, and messages that go to the dmesg log end up. It is usually the first place to look when there is something up and you need to begin investigation.
• log: This log contains messages from the kernel that appears during the boot process, thus it is the first place to look if your system is not behaving properly and you suspect something went wrong during start up, such as a device driver issue, wifi issue or a service that fails to start correctly. For example if you suspect your wifi driver has problems, you might search the dmesg.log file for the string ‘wlan’ or ‘wifi’ like this guy.
• logins/dictionary attacks: If you suspect that there is a security issue with system access you may want to go directly to the ‘auth.log’. The ‘auth.log’ records all user logins and the login mechanism used, and will capture any unauthorised access or attempted access for that matter. For those with servers that have publicly facing IPs this is a log that you should keep an eye on, in particular to monitor dictionary attacks which WILL occur regularly enough.
• tracking sudo access: whenever you run the ‘sudo’ command, it logs it in the auth.log file in case you need a clear picture of who did what, and when 🙂
• log: Diagnosing problems with a new kernel installation (-e.g. distribution upgrade) or major system problems
• apache logs: apache logs can be reviewed for issues with web requests such as broken links (check the apache error log for ‘”file not found’ or ‘404’) or for security issues such as denial of service (DOS) attacks. The apache logs contain the IP address of the machine that made the request, so you can see who is trying to accessing your web application. If someone is performing as DOS attack you can, as a starting point, figure out where this is coming from and if it is a DOS or not. Some apache common error codes are discussed in this post. The full list of possible status codes can be found in the HTTP specification (RFC2616 section 10).
• mysql logs: Database logs can be useful for investigating slow queries or DB related errors. Slow queries are essentially queries in your database that are taking too long. They can result in slow running applications and are one of the most common issues resulting in application performance problems.

rsyslog Daemon – At the heart of the logging mechanism is the rsyslog daemon. This service is responsible for listening to log messages from different parts of a Linux system and routing the message to an appropriate log file in the /var/log directory. It can also forward log messages to another Linux server.

The rsyslog daemon gets its configuration information from the rsyslog.conf file. The file is located under the /etc directory. Basically, the rsyslog.conf file tells the rsyslog daemon where to save its log messages. This instruction comes from a series of two-part lines within the file. This file can be found at rsyslog.d/50-default.conf on ubuntu.

The two part instruction is made up of a selector and an action. The two parts are separated by white space. The selector part specifies what’s the source and importance of the log message and the action part says what to do with the message. The selector itself is again divided into two parts separated by a dot (.). The first part before the dot is called *acility (the origin of the message) and the second part after the dot is called priority (the severity of the message). Together, the facility/priority and the action pair tell rsyslog what to do when a log message matching the criteria is generated. Here is excerpt from a CentOS rsyslog.conf file:

# rsyslog v5 configuration file

…

…

# Include all config files in /etc/rsyslog.d/

IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.* /dev/console

# Log anything (except mail) of level info or higher.

# Don’t log private authentication messages!

*.info;mail.none;authpriv.none;cron.none              /var/log/messages

# The authpriv file has restricted access.

authpriv.*                                             /var/log/secure

# Log all the mail messages in one place.

mail.*                                                 -/var/log/maillog

# Log cron stuff

cron.*                                                 /var/log/cron

# Everybody gets emergency messages

*.emerg                                                 *

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                         /var/log/spooler

# Save boot messages also to boot.log

local7.*                                               /var/log/boot.log

…

The different types of facilities recognized by Linux, are

• auth or authpriv: Messages coming from authorization and security related events
• kern: Any message coming from the Linux kernel
• mail: Messages generated by the mail subsystem
• cron: Cron daemon related messages
• daemon: Messages coming from daemons
• news: Messages coming from network news subsystem
• lpr: Printing related log messages
• user: Log messages coming from user programs
• local0 to local7: Reserved for local use

The list of priorities in ascending order, are

• debug: Debug information from programs
• info: Simple informational message – no intervention is required
• notice: Condition that may require attention
• warn: Warning
• err: Error
• crit: Critical condition
• alert: Condition that needs immediate intervention
• emerg: Emergency condition

So now let’s consider the following line from the file:

cron.*             /var/log/cron

This just tells the rsyslog daemon to save all messages coming from the cron daemon in a file called /var/log/cron. The asterix (*) after the dot (.) means messages of all priorities will be logged. Similarly, if the facility was specified as an asterix, it would mean all sources.

Facilities and priorities can be related in a number of ways. In its default form, when there is only one priority specified after the dot, it means all events equal to or greater than that priority will be trapped. So the following directive causes any messages coming from the mail subsystem with a priority of warning or higher to be logged in a specific file under /var/log:

mail.warn         /var/log/mail.warn

This will log every message equal to or greater than the warn priority, but leave everything below it. So messages with err, crit, alert or emerg will also be recorded in this file. Using an equal sign (=) after the dot (.) will cause only the specified priority to be logged. So if we wanted to trap only the info messages coming from the mail subsystem, the specification would be something like the following:

mail.=info         /var/log/mail.info

Again, if we wanted to trap everything from mail subsystem except info messages, the specification would be something like the following

mail.!info         /var/log/mail.info

or

mail.!=info         /var/log/mail.info

In the first case, the mail.info file will contain everything with a priority lower than info. In the second case, the file will contain all messages with a priority above info. Multiple facilities in the same line can be separated by commas. Multiple sources (facility.priority) in the same line is separated by semicolon. When an action is marked as an asterix (*), it means all users. This entry in my CentOS rsyslog.conf file is saying exactly that:

# Everybody gets emergency messages

*.emerg                                                 *

The configurations for rsyslog can come from other custom files as well. These custom configuration files are usually located in different directories under /etc/rsyslog.d. The rsyslog.conf file includes these directories using $IncludeConfig directive.

Here is what it looks like in Ubuntu:

# Default logging rules can be found in /etc/rsyslog.d/50-default.conf

….

….

$IncludeConfig /etc/rsyslog.d/*.conf

The contents under the /etc/rsyslog.d directory looks like the following:

-rw-r–r– 1 root root 311 Mar 17 2012 20-ufw.conf

-rw-r–r– 1 root root 252 Apr 11 2012 21-cloudinit.conf

-rw-r–r– 1 root root 1655 Mar 30 2012 50-default.conf

Now the destination for a log message does not necessarily have to be a log file; the message can be sent to a user’s console. In this case, the action field will contain the username. If more than one user needs to receive the message, their usernames are separated by commas. If the message needs to be broadcast to every user, it’s specified by an asterix (*) in the action field.

Because of being part of a network operating system, rsyslog daemon can not only save log messages locally, it can also forward them to another Linux server in the network or act as a repository for other systems. The daemon listens for log messages in UDP port 514. The example below will forward kernel critical messages to a server called “texas”.

kern.crit           @texas

logrotate – As more and more information is written to log files, they get bigger and bigger. This obviously poses a potential performance problem. Also, the management of the files become cumbersome.

Linux uses the concept of “rotating” log files instead of purging or deleting them. When a log is rotated, a new log file is created and the old log file is renamed and optionally compressed. A log file can thus have multiple old versions remaining online. These files will go back over a period of time and will represent the backlog. Once a certain number of backlogs have been generated, a new log rotation will cause the oldest log file to be deleted. The rotation is initiated through the logrotate utility. Like rsyslog, logrotate also depends on a configuration file and the name of this file is logrotate.conf. It’s located under /etc.

Here is what I see in the logrotate.conf file

debian@debian:~$ cat /etc/logrotate.conf

# see “man logrotate” for details

# rotate log files weekly

weekly

# keep 4 weeks worth of backlogs

rotate 4

# create new (empty) log files after rotating old ones

create

# uncomment this if you want your log files compressed

#compress

# packages drop log rotation information into this directory

include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here

/var/log/wtmp {

missingok

monthly

create 0664 root utmp

rotate 1

}

/var/log/btmp {

missingok

monthly

create 0660 root utmp

rotate 1

}

# system-specific logs may be configured here

The lines are fairly self-explanatory. By default, log files are to be rotated weekly with four backlogs remaining online at any one time. When the program runs, a new, empty log file will be generated and optionally the old ones will be compressed. The only exception is for wtmp and btmp files. wtmp keeps track of system logins and btmp keeps track of bad login attempts. Both these log files are to be rotated every month and no error is returned if any previous wtmp or btmp file can be found.

Custom log rotation configurations are kept under etc/logrotate.d directory. These are also inluded in the logrotate.conf with the include directive. The Debian installation shows me the content of this directory:

debian@debian:~$ ls -l /etc/logrotate.d

total 44

-rw-r–r– 1 root root 173 Apr 15 2011 apt

-rw-r–r– 1 root root 79 Aug 12 2011 aptitude

-rw-r–r– 1 root root 135 Feb 24 2010 consolekit

-rw-r–r– 1 root root 248 Nov 28 2011 cups

-rw-r–r– 1 root root 232 Sep 19 2012 dpkg

-rw-r–r– 1 root root 146 May 12 2011 exim4-base

-rw-r–r– 1 root root 126 May 12 2011 exim4-paniclog

-rw-r–r– 1 root root 157 Nov 16 2010 pm-utils

-rw-r–r– 1 root root 94 Aug 8 2010 ppp

-rw-r–r– 1 root root 515 Nov 30 2010 rsyslog

-rw-r–r– 1 root root 114 Nov 26 2008 unattended-upgrades

The contents of the rsyslog shows how to recycle a number of log files:

debian@debian:~$ cat /etc/logrotate.d/rsyslog

/var/log/syslog

{

rotate 7

daily

missingok

notifempty

delaycompress

compress

postrotate

invoke-rc.d rsyslog reload > /dev/null

endscript

}

/var/log/mail.info

/var/log/mail.warn

/var/log/mail.err

/var/log/mail.log

/var/log/daemon.log

/var/log/kern.log

/var/log/auth.log

/var/log/user.log

/var/log/lpr.log

/var/log/cron.log

/var/log/debug

/var/log/messages

{

rotate 4

weekly

missingok

notifempty

compress

delaycompress

sharedscripts

postrotate

invoke-rc.d rsyslog reload > /dev/null

endscript

}

As you can see, the syslog file will be reinitialized every day with seven days’ worth of logs being kept online. Other log files are rotated every week. Also worth noting is the postrotate directive. This specifies the action that happens after the whole log rotation has completed.

Logrotate can be manually run to recycle one or more files. And to do that, we simply specify the relevant configuration file as an argument to the command. To see how this works, here is a partial list of log files under /var/log directory in CentOS server:

[root@TestLinux ~]# ls -l /var/log

total 800

…

-rw——- 1 root root   359 Dec 17 18:25 maillog

-rw——-. 1 root root   1830 Dec 16 16:35 maillog-20131216

-rw——- 1 root root 30554 Dec 17 18:25 messages

-rw——-. 1 root root 180429 Dec 16 16:35 messages-20131216

-rw——- 1 root root   591 Dec 17 18:28 secure

-rw——-. 1 root root   4187 Dec 16 16:41 secure-20131216

…

…

The partial contents of the logrotate.conf file looks like this:

[root@TestLinux ~]# cat /etc/logrotate.conf

# see “man logrotate” for details

# rotate log files weekly

weekly

# keep 4 weeks worth of backlogs

rotate 4

# create new (empty) log files after rotating old ones

create

…

…

Next we run the logrotate command:

[root@TestLinux ~]# logrotate -fv /etc/logrotate.conf

Messages scroll over as new files are generated, errors are encountered etc. When the dust settles, we try to check for new mail, secure or messages files:

[root@TestLinux ~]# ls -l /var/log/mail*

-rw——- 1 root root   0 Dec 17 18:34 /var/log/maillog

-rw——-. 1 root root 1830 Dec 16 16:35 /var/log/maillog-20131216

-rw——- 1 root root 359 Dec 17 18:25 /var/log/maillog-20131217

[root@TestLinux ~]# ls -l /var/log/messages*

-rw——- 1 root root   148 Dec 17 18:34 /var/log/messages

-rw——-. 1 root root 180429 Dec 16 16:35 /var/log/messages-20131216

-rw——- 1 root root 30554 Dec 17 18:25 /var/log/messages-20131217

[root@TestLinux ~]# ls -l /var/log/secure*

-rw——- 1 root root   0 Dec 17 18:34 /var/log/secure

-rw——-. 1 root root 4187 Dec 16 16:41 /var/log/secure-20131216

-rw——- 1 root root 591 Dec 17 18:28 /var/log/secure-20131217

[root@TestLinux ~]#

As we can see, all three new log files have been created. The maillog and secure files are still empty, but the new messages file already has some data in it.

Examples

If you’re new to the console and the Linux command line, these commands will get you up and running to the point where you can work with log files at a basic level.

Directory – To change to the log directory, where most of these files sit, use the cd command. This saves having to type out a full path name for every subsequent command – cd /var/log

Editing Files – You can view and edit files in GEdit or Kate, the simple text editors that come with Ubuntu and Kubuntu respectively, but these can be overkill when all you want to do is look at a file or make simple changes. The easiest editor to use from the console is nano, which is less powerful but also less complicated than vim or emacs. The command to edit a particular logfile /var/log/example.log using nano is – nano example.log

Press Ctrl+X to exit. It will ask if you want to save your changes when you exit, but unless you run it with the sudo command the files won’t be writable. In general, you won’t want to save your changes to log files, of course.

Viewing Files – To simply look at a file, an editor is overkill. Use the less command, which pages through a file one screen at a time – less example.log

You don’t need sudo to look at a file. Press h for help, or q to quit. The cursor keys and page up/down keys will work as expected, and the slash key (“/”) will do a case-sensitive search; the n key repeats the last search.

Viewing the Beginning of Files – To see the first ten lines of a file, use the head command:

head example.log

To see some other number of lines from the beginning of the file, add the -n switch, thus:

head -n 20 example.log

Viewing the End of Files – To see the final ten lines of a file, the analogous command is tail:

tail example.log

Again, the -n switch gives you control over how many lines it displays:

tail -n 20 example.log

Watching a Changing File – Also, the -f (“follow”) switch puts tail into a loop, constantly waiting for new additions to the file it’s displaying. This is useful for monitoring files that are being updated in real time:

tail -f example.log

Press Ctrl+C to quit the loop.

Searching Files – Because log files can be large and unwieldy, it helps to be able to focus. The grep command helps you strip out only the content you care about. To find all the lines in a file containing the word “system”, for example, use this:

grep “system” example.log

To find all the lines containing “system” at the beginning of the line, use this:

grep “^system” example.log

The caret symbol, a regular expression that matches only the start of a line. This is less useful for standard log files, which always start with a date and time, but it can be handy otherwise. Not all files have a standard format. Any time the result of a grep is still too long, you can pipe it through less

grep “system” example.log | less

Back to Tutorial