If you want a good overview over your infrastructure but don’t feel the need to install extra software like Logstash, Fluentd or Graylog there is an easy way to achieve a centralized logging facility with what’s already on board in most Linux distributions: rsyslog

This logging daemon is pretty capable and has more features than you might be aware from your distribution’s standard setup. From rsyslog’s homepage:

• Multi-threading
• TCP, SSL, TLS, RELP
• MySQL, PostgreSQL, Oracle and more
• Filter any part of syslog message
• Fully configurable output format
• Suitable for enterprise-class relay chains

To activate the reception of syslog messages from other devices you have two choices. You can opt to send packets via UDP or TCP. UDP lacks TCP’s error checking, so it’s basically fire and forget and you won’t know if your packet arrived at it’s destination.

On the log server

Add the following lines to /etc/rsyslog.conf (old syntax):

# for UDP
$ModLoad imudp
# specify the UDP port to listen on
$UDPServerRun 514

# for TCP
$ModLoad imtcp
# specify the TCP port to listen on
$InputServerRun 514

If you are running a more recent version of rsyslog, you have to use the following syntax:

# for UDP
module(load="imudp")
input(type="imudp" port="514")

# for TCP
module(load="imtcp")
input(type="imtcp" port="514")

Restart your rsyslog daemon with service rsyslog restart or systemctl restart rsyslog and open the port you chose on your firewall for all IPs that will be sending syslog messages to your logging server.

There are ways to define templates for incoming messages, so you can redirect logs of a specific device into subfolders and dynamically create filenames, but I won’t go into them in this article. Maybe there will be a follow-up, that deals with these capabilities of rsyslog.

On the sending device

On the device you want to monitor remotely you have to add one of the following lines to your configuration to start sending all messages to your log server. I like to place this into its own file in /etc/rsyslog.d/90-remote.conf:

# for UDP
*.*   @192.0.2.1:514

# for TCP
*.*   @@192.0.2.1:514

Of course you have to exchange the example IP address with your log server’s address. If you place the remote configuration in its own file, make sure that this file is included in the main rsyslog.conf file (e.g. $IncludeConfig /etc/rsyslog.d/*.conf).

Restart rsyslogd with service rsyslog restart or systemctl restart rsyslog and you should immediately see log messages show up on your log server.

Misc

There are many ways to deal with log messages on the aggregating server, for a relatively small number of devices (<100) you can easily configure rsyslog to put all messages into a MySQL database and either develop your own frontend for searching and filtering or do what I like to do and feed them to LibreNMS with one of the following configuration snippets.

/etc/rsyslog.d/30-librenms.conf (old syntax):

# Feed syslog messages to librenms
$ModLoad omprog
$template librenms,"%FROMHOST%||%syslogfacility-text%||%syslogpriority-text%||%syslogseverity%||%syslogtag%||%$YEAR%-%$MONTH%-%$DAY% %timegenerated:8:25%||%msg%||%programname%\n"

$ActionOMProgBinary /opt/librenms/syslog.php
*.* :omprog:;librenms

/etc/rsyslog.d/30-librenms.conf (new syntax):

# Feed syslog messages to librenms
$ModLoad omprog

$template librenms,"%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n"

:inputname, isequal, "imudp" action(type="omprog"
                                    binary="/opt/librenms/syslog.php"
                                    template="librenms")
& stop