Syslog Integration

Observium supports collection of syslog from devices using rsyslogd or syslog-ng. Syslog capture is achieved by directing the syslog daemon to run Observium's syslog.php script and send syslog messages to it via stdin.

Observium compares the IP address or hostname of the received message to its database to associate it with a device. If no association can be made, the message is discarded. Manual hostname/ip associations can be made via configuration options.

When using Enterprise Edition's distributed and partitioned features, syslog messages can be ingested via syslog.php on any Observium install operating as a poller or web UI.

rsyslogd

Ubuntu 24.04 and greater

Ubuntu 24.04 made changes to AppArmor and are “enforcing” it by default for Rsyslog daemon.

• disable AppArmor enforcing for rsyslogd in Ubuntu 24.04 and greater:

  sudo apt install apparmor-utils
  sudo aa-disable /etc/apparmor.d/usr.sbin.rsyslogd
  

• Check rsyslogd version:

  rsyslogd -v
  

To enable remote logging in rsyslog uncomment or add these two lines in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514

To redirect logs from rsyslog to Observium:

• create file (as root):

      sudo touch /etc/rsyslog.d/30-observium.conf
  

rsyslog 8.xrsyslog 7.x

Add the following to /etc/rsyslog.d/30-observium.conf

#---------------------------------------------------------
# send remote logs to observium

# provides UDP syslog reception
module(load="imudp")

input(type="imudp"
  port="514"
  ruleset="observium")

## provides TCP syslog reception (uncomment if required)
#module(load="imptcp")
#
#input(type="imptcp"
#      port="514"
#      ruleset="observium")

module(load="omprog")

# observium syslog template
template(name="observium"
     type="string"
     string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n")

# observium RuleSets
ruleset(name="observium") {
   action(type="omprog"
          binary="/opt/observium/syslog.php"
          template="observium")
   stop
}

# use this instead if you want filter by severity
#ruleset(name="observium") {
#  if ( $syslogseverity <= '7' ) then {
#    action(type="omprog" binary="/opt/observium/syslog.php" template="observium")
#  }
#}

#---------------------------------------------------------

Severity Level

The example above sets the minimum log severity level to 5, meaning logs of severity 6 and 7 will be discarded.

Add the following to /etc/rsyslog.d/30-observium.conf

#---------------------------------------------------------
#send remote logs to observium

$template observium,"%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg:::space-cc%||%programname%\n"
$ModLoad omprog
$ActionOMProgBinary /opt/observium/syslog.php

:inputname, isequal, "imudp" :omprog:;observium

& ~
# & stop
#---------------------------------------------------------

This enables module omprog, sets the $template like syslog-ng and redirect output to observium's syslog.php.

NOTE, for rsyslog version v7 you may see this warning:

rsyslogd: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]

To solve this you should replace '& ~' with stop:

& stop

• Then restart rsyslog:

      sudo service rsyslog restart
  

• Don't forget to enable syslog in Observium's config.php:

      $config['enable_syslog']                = 1; // Enable Syslog
  

Match syslog hostname/ip with device

• FQDN hostname

NOTE, by default rsyslog uses non-FQDN hostnames, but observium requires FQDN names. To enable FQND hostnames add to the main rsyslog config /etc/rsyslog.conf:

 # Always use full names with domain part
 $PreserveFQDN on

• Match by IP

In case your DNS PTR records don't match actual hostname, or you to want associate devices by IP, you may want to replace the $template variable %fromhost% with %fromhost-ip%.

To correctly matching device by IP, make sure that IPs discovered exist for device on page: device -> Ports -> IPv4 (or IPv6) addresses.

• Mapping unknown hosts

To map unknown syslog hosts with devices, add host_map into your config as in the below example. Where key is syslog unknown host, value is device_id or a hostname known by Observium.

// Mapping (unknown) syslog hosts to device (id or hostname)
$config['syslog']['host_map']['localhost'] = 'my.device.name'; // device hostname/sysname
$config['syslog']['host_map']['127.0.0.1'] = 1;                // or device id

Warning

After changing any rsyslog configs or syslog related configs in Observium you must reload (or restart) the rsyslog service to apply the changes.

sudo service rsyslog reload

Syslog-ng

Make sure these options are set :

 options {
    chain_hostnames(0);
    keep_hostname(1);
    use_dns(no);
 };

Use this as destination in syslog-ng.conf, change syslog.php path to match yours

 source s_net {
    udp();
 };

 destination d_observium { 
    program("/opt/observium/syslog.php" template ("$HOST||$FACILITY||$LEVEL_NUM||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") ); 
 };
 log {
    source(s_net);
    destination(d_observium);
 };

Don't forget to enable syslog in observium config.php:

$config['enable_syslog']                = 1; // Enable Syslog

Restart your syslog-ng server

sudo service syslog-ng restart

Troubleshooting and debugging

To collect raw syslog messages from hosts, follow these steps:

• Add syslog debug to config.php:

  $config['syslog']['debug']      = TRUE;
  

• Make the logs directory writable for everyone (or at least give write permission to the syslog service user):

  sudo chmod o+w /opt/observium/logs
  

• Restart the syslog service:

  sudo service rsyslog restart
  

After that, syslog messages from all hosts will be written to the logs directory (default /opt/observium/logs) with filenames in the format <hostname>.syslog. Wait some time (or simulate syslog messages on the target device) to collect messages from the required host. Don’t forget to disable debugging the same way — either remove or comment out the syslog debug config line and restart the rsyslog service.