Linux snmpd passthrough
The extend directive of Net-SNMP's snmpd agent allows the output of an external command or script to be exposed as SNMP objects. It is described by NET-SNMP-EXTEND-MIB. The agent natively reports only the data it was built to collect, such as CPU load, memory, disks and network interfaces; extend provides a way to publish any other value that a Linux command can produce. This article describes the MIB, the configuration syntax, and common usage patterns.
NET-SNMP-EXTEND-MIB
NET-SNMP-EXTEND-MIB is distributed with Net-SNMP and defines the objects that describe commands configured with extend. Each configured command occupies a row in the MIB's tables, indexed by the string name assigned to it in snmpd.conf. The MIB is located under 1.3.6.1.4.1.8072.1.3, within the enterprise subtree registered to the Net-SNMP project (enterprise number 8072). Walking this OID lists every command currently configured on an agent.
The principal columns are:
- nsExtendOutputFull contains the complete standard output of the command, including line breaks.
- nsExtendOutput1Line contains only the first line of output, and is suited to commands that return a single value or short status string.
- nsExtendOutputNumLines contains the number of lines of output.
- nsExtendResult contains the exit status of the command as an integer.
All of these objects are read-only. The agent does not cache their values by default; the command is executed again when an object is requested (see "Reducing the cost of slow commands" below).
Comparison with the exec directive
Earlier versions of Net-SNMP provided a similar exec directive. Its output is published in the extTable of UCD-SNMP-MIB, where rows are identified by a numeric index rather than by name and fewer attributes are available. Although exec remains supported for compatibility, extend is the documented mechanism for new configurations and is the one described here.
Writing a script
Any executable file can be used, including shell scripts and programs written in languages such as Python or Perl. The agent runs the command directly rather than through a shell, so a script requires a shebang line and execute permission. Only standard output is captured; text written to standard error is not published.
The following script reports the number of messages in a Postfix mail queue, a value that snmpd does not collect itself:
#!/bin/sh
# /usr/local/bin/check_mailq.sh
mailq | grep -c "^[A-F0-9]"
chmod +x /usr/local/bin/check_mailq.sh
A script should be run manually, under the account that snmpd uses, before it is added to the configuration, since a command that fails when run directly will also fail when run by the agent.
Configuration
An entry is created by a single line in /etc/snmp/snmpd.conf consisting of the directive, a name, the command, and any fixed arguments:
# /etc/snmp/snmpd.conf
extend mailqueue-depth /usr/local/bin/check_mailq.sh
sudo systemctl restart snmpd
The command must be given as an absolute path, because the agent does not search the $PATH environment variable. Because the command is not interpreted by a shell, constructs such as pipes and redirection are unavailable unless a shell is invoked explicitly:
extend disk-report /bin/sh -c "df -h / | tail -1"
Querying the results
Each entry is indexed by its configured name, which is written as a quoted string when a single value is requested:
snmpget -v2c -c public localhost NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."mailqueue-depth"
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."mailqueue-depth" = STRING: 4
All configured entries can be retrieved by walking the output table:
snmpwalk -v2c -c public localhost NET-SNMP-EXTEND-MIB::nsExtendOutputTable
If a new entry returns no data or the response "No such instance", the usual cause is that the agent was unable to execute the command. Possible causes are listed under "Permissions and common problems" below.
Exit status and alerting
Because nsExtendResult holds the exit status of the command as an integer, a script can signal its state without any text being parsed by the monitoring system. The convention used by Nagios-style check plugins is to return 0 for a normal condition and a non-zero value for a fault:
#!/bin/sh
# /usr/local/bin/check_myapp.sh
SERVICE="myapp.service"
if systemctl is-active --quiet "$SERVICE"; then
echo "OK: $SERVICE is running"
exit 0
else
echo "CRITICAL: $SERVICE is not running"
exit 2
fi
extend myapp-health /usr/local/bin/check_myapp.sh
snmpget -v2c -c public localhost NET-SNMP-EXTEND-MIB::nsExtendResult."myapp-health"
NET-SNMP-EXTEND-MIB::nsExtendResult."myapp-health" = INTEGER: 2
The accompanying text is available in nsExtendOutput1Line for display to an operator.
Reusing one script for several targets
A single script that accepts arguments can be referenced by several extend lines. The entry name distinguishes the resulting rows:
#!/bin/sh
# /usr/local/bin/check_disk.sh - prints the usage percentage of the mount point given as $1
df -h "$1" | awk 'NR==2 {print $5}'
# /etc/snmp/snmpd.conf
extend disk-root /usr/local/bin/check_disk.sh /
extend disk-var /usr/local/bin/check_disk.sh /var
extend disk-data /usr/local/bin/check_disk.sh /data
Each entry appears as a separate row, for example nsExtendOutput1Line."disk-root" and nsExtendOutput1Line."disk-var".
Reducing the cost of slow commands
Since the command is run each time its output is requested, a command that takes a long time to complete delays every poll of the corresponding object, and can delay other responses from the agent. A common approach is to run the slow task separately, on a schedule, and have the agent read the stored result. In the following example, cron runs the check every five minutes and writes its output to a file, and the extend entry only reads that file:
# crontab
*/5 * * * * /usr/local/bin/expensive_check.sh > /var/cache/snmp-myextend.txt 2>&1
# /etc/snmp/snmpd.conf
extend myextend /bin/cat /var/cache/snmp-myextend.txt
The response time of each SNMP request is then independent of the duration of the underlying check.
Permissions and common problems
- Paths. Both the script and any programs it calls should be referenced by absolute path.
- Account. The agent runs under a particular user, which can be identified with
ps -o user= -C snmpd. Testing the script as that user (sudo -u <user> /path/to/script.sh) identifies the most frequent cause of failure, in which a script works for an administrator but the agent's account lacks permission to read a file, access the network, or execute a dependency. - Privileges. Granting the agent's account broad
sudoor root access is discouraged, because any host permitted to query the agent can cause the script to run. Where elevated privileges are required, they are normally limited to a single command through a specificsudoersentry. - Output stream. Only standard output is captured; standard error is discarded.
- Reloading. Changes to
snmpd.conftake effect after the agent is restarted.
Alternatives: pass and pass_persist
Objects created with extend always appear in the generic NET-SNMP-EXTEND-MIB tree and are indexed by name. This is unsuitable where data must be placed in a specific OID subtree, such as one under a private enterprise number, or where table traversal (GETNEXT) must be implemented by the handler. For these cases Net-SNMP provides the pass and pass_persist directives, which delegate an entire OID subtree to an external program that implements the SNMP request handling itself. They are described in the Using net-snmp guide.