If you want a good overview over your infrastructure but don’t feel the need to install extra software like Logstash, Fluentd or Graylog there is an easy way to achieve a centralized logging facility with what’s already on board in most Linux distributions: rsyslog
This logging daemon is pretty capable and has more features than you might be aware from your distribution’s standard setup. From rsyslog’s homepage:
- TCP, SSL, TLS, RELP
- MySQL, PostgreSQL, Oracle and more
- Filter any part of syslog message
- Fully configurable output format
- Suitable for enterprise-class relay chains
To activate the reception of syslog messages from other devices you have two choices. You can opt to send packets via UDP or TCP. UDP lacks TCP’s error checking, so it’s basically fire and forget and you won’t know if your packet arrived at it’s destination.
On the log server
Add the following lines to
/etc/rsyslog.conf (old syntax):
# for UDP $ModLoad imudp # specify the UDP port to listen on $UDPServerRun 514 # for TCP $ModLoad imtcp # specify the TCP port to listen on $InputServerRun 514
If you are running a more recent version of rsyslog, you have to use the following syntax:
# for UDP module(load="imudp") input(type="imudp" port="514") # for TCP module(load="imtcp") input(type="imtcp" port="514")
Restart your rsyslog daemon with
service rsyslog restart or
systemctl restart rsyslog and open the port you chose on your firewall for all IPs that will be sending syslog messages to your logging server.
There are ways to define templates for incoming messages, so you can redirect logs of a specific device into subfolders and dynamically create filenames, but I won’t go into them in this article. Maybe there will be a follow-up, that deals with these capabilities of rsyslog.
On the sending device
On the device you want to monitor remotely you have to add one of the following lines to your configuration to start sending all messages to your log server. I like to place this into its own file in
# for UDP *.* @192.0.2.1:514 # for TCP *.* @@192.0.2.1:514
Of course you have to exchange the example IP address with your log server’s address. If you place the remote configuration in its own file, make sure that this file is included in the main
rsyslog.conf file (e.g.
Restart rsyslogd with
service rsyslog restart or
systemctl restart rsyslog and you should immediately see log messages show up on your log server.
There are many ways to deal with log messages on the aggregating server, for a relatively small number of devices (<100) you can easily configure rsyslog to put all messages into a MySQL database and either develop your own frontend for searching and filtering or do what I like to do and feed them to LibreNMS with one of the following configuration snippets.
/etc/rsyslog.d/30-librenms.conf (old syntax):
# Feed syslog messages to librenms $ModLoad omprog $template librenms,"%FROMHOST%||%syslogfacility-text%||%syslogpriority-text%||%syslogseverity%||%syslogtag%||%$YEAR%-%$MONTH%-%$DAY% %timegenerated:8:25%||%msg%||%programname%\n" $ActionOMProgBinary /opt/librenms/syslog.php *.* :omprog:;librenms
/etc/rsyslog.d/30-librenms.conf (new syntax):
# Feed syslog messages to librenms $ModLoad omprog $template librenms,"%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n" :inputname, isequal, "imudp" action(type="omprog" binary="/opt/librenms/syslog.php" template="librenms") & stop