Network Management in remote destinations

Our target is to monitor multiple Natrix Gateway from a single OpenNMS instance. Most likely all these instances are running in isolated networks and are hard to reach from this instance. It would require building complex Firewall rules between this instance and the isolated networks to allow SNMP and other traffic going back and forth between the OpenNMS Server and the various Natrix Gateways.

We are leveraging OpenNMS Minions to overcome this challenge. The Minions can operate behind firewall and/or NAT, if they can communicate with OpenNMS via REST (backed by Jetty) and JMS (provided by an embedded ActiveMQ broker).

OpenNMS Minions

OpenNMS Minion is an instance of the Karaf OSGi service that can run distributed OpenNMS components that are packaged as Karaf features.

OpenNMS can associate nodes to Locations to manage them geographically. This is accomplished by the OpenNMS Provisioning System. Every node created in OpenNMS is by default assigned to the Location Default.  That means that all nodes which belong to the Default Location are managed by the OpenNMS instance itself. For each Natrix Controller which runs in a branch a separate Location is needed. This also allows you to manage all foreign nodes in this Location. The Location is a configuration property of the Minion as we see later in this chapter.

On startup, Minions register themselves automatically to the OpenNMS instance. OpenNMS delegates monitoring requests of Nodes which belong to a Minion Location to the registered Minions and uses these Minions as a kind of proxy service. Figure:

Before you begin

Managing the Minion user

On your OpenNMS instance, create a user Minion and a password for this user. The user Minion requires the role ROLE_MINION.

Setting up an SNMP Agent on the Natrix Controller

Setting up a SNMP agent on the Natrix Controller is a quite simple task.

As always, it is recommended to update existing packages before installing any new packages:

apt-get update

Afterwards, you can install the three required packages for a fully functional SNMP agent on your Natrix Controller. These are

• the SNMP Agent snmpd
• the SNMP Manager snmp
• the MIBs downloader snmp-mibs-downloader

apt-get install snmpd snmp snmp-mibs-downloader

In case the MIBS downloader doesn’t automatically download the MIBs database, you can initiate this manually:

download-mibs

To use OID names instead of the cryptic 1.3.x.x.x.x. numbers, it is necessary to comment out the ‘mibs’ string in the file /etc/snmp/snmp.conf.

# As the snmp packages come without MIB files due to license reasons, loading
# of MIBs is disabled by default. If you added the MIBs you can reenable
# loading them by commenting out the following line.
###mibs :

We will use the procfix parameter to enable modifications on the service status of a process (stop, start, restart) and hence we require an additional setting in the /etc/sudoers file:

### for snmp procfix scripts as root
snmp ALL=(ALL) NOPASSWD: ALL 

The SNMP service runs by default under user: snmp and group: snmp. This user may not start or end any processes, and certainly not restart the entire controller. To make this possible, the user snmp must be assigned root rights via sudo. The entry shown is necessary for this. After each change, the snmpd service must be restarted to activate the setting.

This is it. We should take care that the SNMP daemon starts after any boot, and starting the daemon for the first time:

systemctl enable snmpd
service snmpd start

Configuring and testing the SNMP Agent on the Natrix Controller

The main configuration happens in the /etc/snmp/snmpd.conf file

The configuration file has several sections which we want to go through section by section, followed with some test after each configuration to check accuracy of our configuration changes done. We are concentrating on the most relevant parameters only, and leave less used parameters out to reduce complexity and to increase readability of the configuration chapter.

In terms of testing, there are two useful command-line SNMP tools available after the installation: snmpwalk and snmpget.

Snmpget returns for a given resource (Server, PC, etc) the current value for a specific OID back

Syntax:

snmpget -v1 -c <Community-String> <IP-Adresse> <OID of resource>

-	you start with the SNMP version to be used for this request. Options are V1, V2C or V3
-	the you determine the Community String which stands for authentication
-	 the host you would send the query to
-	The particular resource of a MIB you sent the query for

A query (and output) of complete MIBs can be carried out with snmpwalk. With snmpwalk, tables and sub nodes of an OID node can be searched easily.

Syntax:

snmpwalk -v1 -c <Community-String> <IP-Adresse> <OID of MIB>

-	you start with the SNMP version to be used for this request. Options are V1, V2C or V3
-	the you determine the Community String which stands for authentication
-	 the host you would send the query to
-	The entire MIB resource you sent the query for

The Agent Behaviour configuration section

Here we are defining a list of listening addresses on which to receive incoming SNMP requests.

This can be multiple ones (localhost, eth0, wlan0, etc), a particular one (localhost only), can be done via UDP or TCP, via IPV4 or IPV6.

Best practice is to allow SNMP traffic on all interfaces, regardless if they bind with the IPV4 or IPV6 stack, but on UDP and using the standard port 161:

The SNMPV3 configuration section

Security functions were not sufficiently available up to and including SNMP v2, therefore SNMP v3 was specified. It increases security enormously through security features such as username and password, as well as data transmission encryption. The complexity increases to the same extent, of course, there are significantly more configuration options available compared to v2c.

We will concentrate on data security in a later chapter; hence we leave the configuration for V3 out here. Let’s work with v2c going forward until we reach the point to ‘harden’ our Natrix controller.

SNMPv3 AUTHENTICATION
#
#  Note that these particular settings don’t actually belong here.
#  They should be copied to the file /var/lib/snmp/snmpd.conf
#     and the passwords changed, before being uncommented in that file *only*.
#  Then restart the agent

#  createUser authOnlyUser  MD5 “remember to change this password”
#  createUser authPrivUser  SHA “remember to change this one too”  DES
#  createUser internalUser  MD5 “this is only ever used internally, but #still change the password”

#  If you also change the usernames (which might be sensible),
#  then remember to update the other occurances in this example config file #  to match.

The Access Control configuration section

The SNMPD daemon offers four distinct methods of controlling which users have which kinds of access and to which parts of the MIB tree. Traditionally the community-based access control was the only supported access control method available when using SNMPv1 or SNMPv2c. This changed with SNMPV3 which introduced the concept of users to the traditional access control model. Unlike community identifiers, which are usually identical across clients and transmitted in clear-text, users are intended to be unique and can make use of advanced authentication features and optional encryption.

But coming back to SNMPV1 and SNMPV2c. These versions provide simple access control based around the concept of a community identifier. Clients which connect using community-based authentication are required to provide a community identifier which is then used to allow or deny access depending on the configuration.

As the community identifier is usually identical for all clients and is transmitted in plain-text over the network this authentication method only offers a bare minimum level of security and should therefore never be used over the public Internet or on an untrusted network, especially to provide write access. As already discussed, we just briefly touch this concept as we want to move to the more secure way to control access via user credentials in SNMPV3, and what we will discuss in a later chapter.

Let’s start with the read-only community, represented by the rocommunity parameter. There is a default community ‘public’ available right after install.

The syntax to declare communities within the SNMPD configuration file is as follows:

IPV4:    rocommunity  COMMUNITY [SOURCE [OID]]

IPV6:    rocommunity6 COMMUNITY [SOURCE [OID]]

It’s obvious that the community identifier is only required parameter which must be supplied by clients for authentication purposes. However, access can be further limited by specifying a source address of the client submitting the request (consider that whenever you read ‘default’ as source address, ‘default’ represents localhost). This mask restricts clients to a particular network address. Furthermore, an OID can also be specified to apply access restriction to a single branch of the MIB tree. You can combine both parameters if required to express the full range of authentication options. Caution: the first entry which matches the client request will be used! It is not required to have both fulfilled.

You can assign views to community strings to limit which MIB objects a user can access. You can use a predefined view or create your own view. So instead of assigning an OID to a rocommunity string, you can also define a view which includes certain MIBS to be accessed and restrict access to this view only by assigning this view to this specific rocommunity. Let’s do this!

So, in our case for the Natrix controller, we define a view called ‘systemonly’  and a community called ‘natrixPUB’ .

To make the ‘natrixPUB’ available on all IP interfaces, we define it in the following way:

rocommunity natrixPUB 0.0.0.0/0 -V systemonly

If you for example want to restrict access to the view ‘systemonly’ to users which are accessing the SNMP agent from a specific network segment, then you can do this the following way:

rocommunity natrixPUB 192.168.188.0/24 -V systemonly

Now it comes to the definition of what kind of information everybody belonging to the ‘natrixPUB’ is able to query for. As we already said, we want to configure this on the basis of ‘views’.

Let’s remember that MIBS represent a catalogue of different OIDs which all belong to the same entity in a network. ‘Views’ can control access on both MIB – or on  OID level.

For the Natrix controller we would like to monitor the following resources on our Raspberry PI:

1. System SNMP variables such as description, location, SNMP uptime: 1.3.6.1.2.1.1
2. System uptime:                                                                                     1.3.6.1.2.1.25.1
3. Host ResourcesMIB-useful host information (HD, Memory, etc):         1.3.6.1.2.1.25
4. UCD- System MIB for monitoring disks, memory, etc:                          1.3.6.1.4.1.2021

#
#  ACCESS CONTROL
#

#  system + hrSystem groups only
view   systemonly  included   .1.3.6.1.2.1.1
view   systemonly  included   .1.3.6.1.2.1.25.1
view systemonly included 	  .1.3.6.1.2.1.25
view   systemonly  included   .1.3.6.1.4.1.2021


#<<<<<<<IF-MIB This is required for SNMP Interfaces
view   systemonly  included   .1.3.6.1.2.1.2

#<<<<< HOST-RESOURCES-MIB
view   systemonly  included   .1.3.6.1.2.1.25 
#<<<<< UCD-SNMP-MIB
view   systemonly  included   .1.3.6.1.4.1.2021  

#<<<<< prTableMonitor
view   systemonly  included   .1.3.6.1.4.1 

#<<<<< Version Configuration Options OID
view   	systemonly  included	1.3.6.1.4.1.2021.100.6.0

view all    included  .1                               80

#
# Community string
#
rocommunity natrixPUB   - V systemonly
rwcommunity natrixPRIV   - V systemonly

#  Full access from the local host
rocommunity natrixPUB 127.0.0.1 .1 - V systemonly

#  Default access to basic system info
 rocommunity natrixPUB  default    -V systemonly

#  rocommunity6 is for IPv6
 rocommunity6 natrixPUB  default   -V systemonly

With the configuration of the Agent Behaviour and the Access Control we can run our first test to see if our configuration is working.

Please don’t forget to restart the SNMP daemon after each configuration change in the /etc/snmp/snmpd.conf file:

service snmpd restart

For running the test, we have three commands available which can be executed on the console:

• snmpwalk: Read access to SNMP variables with all values below in the tree (typically used to query for entire MIBS).
• snmpget: Read access to only one single OID value.
• snmpwrite: Write access to a value.

With snmpwalk we can see all value of a specific tree, so for example for the SNMP MIB-2 System:

snmpwalk -v1  -c natrixPUB   127.0.0.1 1.3.6.1.2.1.1

results in the following output:

SNMPv2-MIB::sysDescr.0 = STRING: Linux openHABianDevice 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (469552) 1:18:15.52
SNMPv2-MIB::sysContact.0 = STRING: Natrix Admin (192.168.188.22) <info@converge-it.de>  Version 1.0 from December 21st 2020
SNMPv2-MIB::sysName.0 = STRING: NatrixController
SNMPv2-MIB::sysLocation.0 = STRING: 777 Natrix Avenue , Natrixtown, Natrixcountry
SNMPv2-MIB::sysServices.0 = INTEGER: 72
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORID.1 = OID: SNMP-MPD-MIB::snmpMPDCompliance
SNMPv2-MIB::sysORID.2 = OID: SNMP-USER-BASED-SM-MIB::usmMIBCompliance
SNMPv2-MIB::sysORID.3 = OID: SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance
SNMPv2-MIB::sysORID.4 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.5 = OID: SNMP-VIEW-BASED-ACM-MIB::vacmBasicGroup
SNMPv2-MIB::sysORID.6 = OID: TCP-MIB::tcpMIB
SNMPv2-MIB::sysORID.7 = OID: IP-MIB::ip
SNMPv2-MIB::sysORID.8 = OID: UDP-MIB::udpMIB
SNMPv2-MIB::sysORID.9 = OID: SNMP-NOTIFICATION-MIB::snmpNotifyFullCompliance
SNMPv2-MIB::sysORID.10 = OID: NOTIFICATION-LOG-MIB::notificationLogMIB
SNMPv2-MIB::sysORDescr.1 = STRING: The MIB for Message Processing and Dispatching.
SNMPv2-MIB::sysORDescr.2 = STRING: The management information definitions for the SNMP User-based Security Model.
SNMPv2-MIB::sysORDescr.3 = STRING: The SNMP Management Architecture MIB.
SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module for SNMPv2 entities
SNMPv2-MIB::sysORDescr.5 = STRING: View-based Access Control Model for SNMP.
SNMPv2-MIB::sysORDescr.6 = STRING: The MIB module for managing TCP implementations
SNMPv2-MIB::sysORDescr.7 = STRING: The MIB module for managing IP and ICMP implementations
SNMPv2-MIB::sysORDescr.8 = STRING: The MIB module for managing UDP implementations
SNMPv2-MIB::sysORDescr.9 = STRING: The MIB modules for managing SNMP Notification, plus filtering.
SNMPv2-MIB::sysORDescr.10 = STRING: The MIB module for logging SNMP Notifications.
SNMPv2-MIB::sysORUpTime.1 = Timeticks: (7) 0:00:00.07
SNMPv2-MIB::sysORUpTime.2 = Timeticks: (7) 0:00:00.07
SNMPv2-MIB::sysORUpTime.3 = Timeticks: (7) 0:00:00.07
SNMPv2-MIB::sysORUpTime.4 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.5 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.6 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.7 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.8 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.9 = Timeticks: (8) 0:00:00.08
SNMPv2-MIB::sysORUpTime.10 = Timeticks: (8) 0:00:00.08

snmpwalk -v1  -c natrixPUB   127.0.0.1 1.3.6.1.2.1.1

With snmpget we can now query for a specific OID, so for example for the system description OID (sysDescr .1.3.6.1.2.1.1.1.0):

snmpget -v1  -On -c natrixPUB   127.0.0.1 .1.3.6.1.2.1.1.1.0

results in the following output:

.1.3.6.1.2.1.1.1.0 = STRING: Linux NatrixController 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l

If you analyze the snmpget request carefully, you can see that we have added on additional option -On . This option converts the OID names into the OID numbering format.

Same snmpget request without the -On option results in the following output:

snmpget -v1  -c natrixPUB   127.0.0.1 .1.3.6.1.2.1.1.1.0

results in the following output:

SNMPv2-MIB::sysDescr.0 = STRING: Linux NatrixController 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l

Some more useful snmpwalk / snmpget queries:

Filesystems

snmpwalk -v2c -c natrixPUB -On localhost .1.3.6.1.2.1.25.2.3

Systeminfo

snmpget -v2c -c natrixPUB localhost .1.3.6.1.2.1.1.1.0
snmpget -v2c -c natrixPUB localhost .1.3.6.1.2.1.1.4.0
snmpget -v2c -c natrixPUB  localhost .1.3.6.1.2.1.1.6.0

Uptime

snmpget -v2c -c natrixPUB localhost .1.3.6.1.2.1.25.1.1.0

Network / data transfer

snmpwalk -v2c -c natrixPUB localhost .1.3.6.1.2.1.2.2.1
snmpwalk -v2c -c natrixPUB -On localhost .1.3.6.1.2.1.2.2.1

Processes

snmpwalk -v2c -c natrixPUB localhost .1.3.6.1.2.1.25.4.2.1

The System Configuration section

In the System Configuration section, you can provide some details to the location of the system (.1.3.6.1.2.1.1.6.0) or a contact person for the system (.1.3.6.1.2.1.1.4.0). 

#
#  SYSTEM INFORMATION
#  Note that setting these values here, results in the corresponding #MIB objects being ‘read-only’
#  See snmpd.conf(5) for more details
sysLocation    777 Natrix Avenue , Natrixtown, Natrixcountry
sysContact     Natrix Admin (192.168.188.22) <info@converge-it.de>  Version 1.0 from December 21st 2020

The Process Monitoring Configuration section

The hrSWRun group of the Host Resources MIB (.1.3.6.1.2.1.25) provides information about individual processes running on the local system. The ystemo of the UCD-SNMP-MIB (.1.3.6.1.4.1.2021)  complements this by reporting on selected services (which may involve multiple processes).

The ‘proc’ parameter monitors processes with the given name (based on the output of /bin/ps -e). If the number of named processes is less than MIN or greater than MAX, then the corresponding prErrorFlag instance will be set to 1, and a suitable description message reported via the prErrMessage instance If we take the default settings in the /etc/snmpd.conf file 

#
#  Process Monitoring
#
# At least one  ‘mountd’ process
proc  mountd
# No more than 4 ‘ntalkd’ processes – 0 is OK
proc  ntalkd    4
# At least one ‘sendmail’ process, but no more than 10
proc  sendmail 10 1

#  init process dummy for reboot
proc    init                
procfix init /usr/bin/sudo /sbin/reboot

#  process ser2net / at least one   process
proc    ser2net                     
procfix ser2net  /usr/bin/sudo  /etc/init.d/ser2net restart  


#  Walk the UCD-SNMP-MIB:: ystem o to see the resulting output
#  Note that this table will be empty if there are no “proc” entries in # the snmpd.conf file

Then we will get the following output when walking through the UCD-SNMP-MIB::portable:

snmpwalk -v1  -c natrixPUB   127.0.0.1 UCD-SNMP-MIB:: ystem o
UCD-SNMP-MIB::prIndex.1 = INTEGER: 1
UCD-SNMP-MIB::prIndex.2 = INTEGER: 2
UCD-SNMP-MIB::prIndex.3 = INTEGER: 3
UCD-SNMP-MIB::prIndex.4 = INTEGER: 4
UCD-SNMP-MIB::prIndex.5 = INTEGER: 5
UCD-SNMP-MIB::prNames.1 = STRING: mountd
UCD-SNMP-MIB::prNames.2 = STRING: ntalkd
UCD-SNMP-MIB::prNames.3 = STRING: sendmail
UCD-SNMP-MIB::prNames.4 = STRING: init
UCD-SNMP-MIB::prNames.5 = STRING: ser2net
UCD-SNMP-MIB::prMin.1 = INTEGER: 1
UCD-SNMP-MIB::prMin.2 = INTEGER: 0
UCD-SNMP-MIB::prMin.3 = INTEGER: 1
UCD-SNMP-MIB::prMin.4 = INTEGER: 1
UCD-SNMP-MIB::prMin.5 = INTEGER: 1
UCD-SNMP-MIB::prMax.1 = INTEGER: 0
UCD-SNMP-MIB::prMax.2 = INTEGER: 4
UCD-SNMP-MIB::prMax.3 = INTEGER: 10
UCD-SNMP-MIB::prMax.4 = INTEGER: 0
UCD-SNMP-MIB::prMax.5 = INTEGER: 0
UCD-SNMP-MIB::prCount.1 = INTEGER: 0
UCD-SNMP-MIB::prCount.2 = INTEGER: 0
UCD-SNMP-MIB::prCount.3 = INTEGER: 0
UCD-SNMP-MIB::prCount.4 = INTEGER: 1
UCD-SNMP-MIB::prCount.5 = INTEGER: 0
UCD-SNMP-MIB::prErrorFlag.1 = INTEGER: error(1)
UCD-SNMP-MIB::prErrorFlag.2 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.3 = INTEGER: error(1)
UCD-SNMP-MIB::prErrorFlag.4 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.5 = INTEGER: error(1)
UCD-SNMP-MIB::prErrMessage.1 = STRING: No mountd process running
UCD-SNMP-MIB::prErrMessage.2 = STRING:
UCD-SNMP-MIB::prErrMessage.3 = STRING: No sendmail process running
UCD-SNMP-MIB::prErrMessage.4 = STRING:
UCD-SNMP-MIB::prErrMessage.5 = STRING: No ser2net process running
UCD-SNMP-MIB::prErrFix.1 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFix.2 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFix.3 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFix.4 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFix.5 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFixCmd.1 = STRING:
UCD-SNMP-MIB::prErrFixCmd.2 = STRING:
UCD-SNMP-MIB::prErrFixCmd.3 = STRING:
UCD-SNMP-MIB::prErrFixCmd.4 = STRING: /usr/bin/sudo /sbin/reboot
UCD-SNMP-MIB::prErrFixCmd.5 = STRING: /usr/bin/sudo  /etc/init.d/ser2net restart

What we see in the output is the following:

• 5 processes defined to be monitored by the SNMP Agent
• These are: mountd, ntalkd, sendmail, init and ser2net
• Min/Max values for e.g. sendmail (index 3) is 1(min) and 10 (max)
• Only ntalkd (index 2) and init (index 4) are running
• Two process fixes are defined. /usr/bin/sudo /sbin/reboot for init (index 4) and  /usr/bin/sudo for ser2net (index 5)

Let’s monitor our Mosquitto service and restart it on failure.

For a better readability, I will leave most of the other processes out. For us it doesn’t make much sense to monitor processes such as sendmail, mountd, ntalkd or ser2net, but init for remotely restarting the whole system might be useful at a certain point of time.

For restarting purposes, we define the procfix parameter. Procfix registers a command that can be run to fix errors with the given process NAME. This will be invoked when the corresponding prErrFix instance is set to 1.

The new Process Monitoring section looks like this now after the modification:

#
#  Process Monitoring
#

#  init process dummy for reboot
proc    init                
procfix init /usr/bin/sudo /sbin/reboot

#  process Mosquitto / at least one   process
#  init process dummy for reboot
proc    init                
procfix init /usr/bin/sudo /sbin/reboot

#  process ser2net / at least one   process
proc    ser2net                     
procfix ser2net  /usr/bin/sudo  /etc/init.d/ser2net restart    


#  Walk the UCD-SNMP-MIB::prTable to see the resulting output
#  Note that this table will be empty if there are no "proc" entries in #  the snmpd.conf file

According to those changes, we will get the following output when walking through the UCD-SNMP-MIB::portable:

snmpwalk -v1  -c natrixPUB   127.0.0.1 UCD-SNMP-MIB::prTable
UCD-SNMP-MIB::prIndex.1 = INTEGER: 1
UCD-SNMP-MIB::prIndex.2 = INTEGER: 2
UCD-SNMP-MIB::prNames.1 = STRING: init
UCD-SNMP-MIB::prNames.2 = STRING:  ystem o
UCD-SNMP-MIB::prMin.1 = INTEGER: 1
UCD-SNMP-MIB::prMin.2 = INTEGER: 1
UCD-SNMP-MIB::prMax.1 = INTEGER: 0
UCD-SNMP-MIB::prMax.2 = INTEGER: 0
UCD-SNMP-MIB::prCount.1 = INTEGER: 1
UCD-SNMP-MIB::prCount.2 = INTEGER: 1
UCD-SNMP-MIB::prErrorFlag.1 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.2 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrMessage.1 = STRING:
UCD-SNMP-MIB::prErrMessage.2 = STRING:
UCD-SNMP-MIB::prErrFix.1 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFix.2 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrFixCmd.1 = STRING: /usr/bin/sudo /sbin/reboot
UCD-SNMP-MIB::prErrFixCmd.2 = STRING: /usr/bin/sudo  /etc/init.d/ ystem o restart

What we see is the following:

• Our mosquito service has the index 2:

UCD-SNMP-MIB::prNames.2 = STRING: mosquito

• Our mosquito service has no error:

UCD-SNMP-MIB::prErrorFlag.2 = INTEGER: noError(0)

• To restart the mosquito service (manual or on error):

UCD-SNMP-MIB::prErrFixCmd.2 = STRING: /usr/bin/sudo  /etc/init.d/ystemo restart

To initiate a restart, we need to set the value ‘1’ for the error flag on UCD-SNMP-MIB::prErrorFlag.2

snmpset -v2c -c writesecret localhost UCD-SNMP-MIB::prErrorFlag.2 ™ 1

Minions installation on Natrix Controller

For the installation there are some things to be considered:

• The OpenNMS instance version must be the same version as the Minion packages
• ReST and ActiveMQ to the OpenNMS instance must be possible
  • The OpenNMS ReST Interface is running on port 8980 by default
  • The ActiveMQ broker used by OpenNMS is running on port 61616 by default.

The Minion currently does support the following Management Protocols :

• Syslog – Receiving Syslog messages and proxying them through ActiveMQ to the OpenNMS instance
• SNMP Traps –  Receiving SNMPT Traps and proxying them through ActiveMQ to the OpenNMS instance
• SNMP Performance data collections
• Service Monitors – testing availability of- and measuring response times from applications and proxying this information through ActiveMQ to the OpenNMS instance

We all will further describe this in a later chapter.

Adding apt Repository

Create and edit a file opennms.list  in the folder /etc/apt/sources.list.d

echo ‘deb https://debian.opennms.org opennms-28 main \
      deb-src https://debian.opennms.org opennms28 main’ > /etc/apt/sources.list.d/opennms.list

Note: This is currently the latest Minion version available. Although it doesn’t match the version of the OpenNMS instance (26.2.2), it works fine.

Download the key and run apt update:

wget -O – https://debian.opennms.org/OPENNMS-GPG-KEY |apt-key add –
apt update

Now, you are ready to install the Minion package

apt -y install opennms-minion

If you want to start Minion after each system boot, just add the following line:

systemctl enable minion

Workaround to eliminate JNI incompatibility on our Natrix Gateway

There is an issue reported with a 32-bit operating system. This issue as that the Minion application runs into a problem with Java Native Access (JNA) which is being used for ICMP. (GitHub(1), 2021).

The workaround is documented here. (openNMS(1), 2021)

We go for the option to substitute the corresponding two files with an ewer Version of the JNDI but keep the name. JNA 4.4.0 has an issue with an ARM -32 bit system. This has been resolved with JNA 5.5.0 files required are jna-5.5.0.jar and jna-platform-5.5.0.jar.

The following procedure needs to be executed:

/usr/share/minion/system/net/java/dev/jna/jna-platform/4.4.0, copied jna-platform-5.5.0.jar into that directory and renamed it to jna-platform-4.4.0.jar

/usr/share/minion/system/net/java/dev/jna/jna/4.4.0, copied jna-5.5.0.jar into that directory and renamed it to jna-4.4.0.jar

That’s it!

The following directory structure has been setup by the Minion package:

tree -L 1 /usr/share/minion

/usr/share/minion
├── bin
├── data -> /var/lib/minion/data
├── deploy -> /var/lib/minion/deploy
├── etc -> /etc/minion
├── instances
├── karaf.pid
├── lib
├── lock
├── repositories
└── system

Minions configuration on Natrix Gateway

When Minion starts up, it automatically connects to the OpenNMS instance. So it requires some information to initiate the RestFUL API calls and to set a an ActiveMQ JMS session.

This is done in the file  /etc/minion/ org.opennms.minion.controller.cfg

nano /etc/minion/org.opennms.minion.controller.cfg

location = NatrixGateway1
id = 1cad9928-f691-11ea-8706-dca63298d890
http-url http://<your-opennms-instance>:8980/opennms
broker-url failover:tcp:// <your-opennms-instance>:61616

• The location parameter is a free text representing the Location name which appears in the OpenNMS instance for this Minion branch.
• The id is a random number generated by Minion when the package is installed.
• The  http-url parameter represents the RestFUL API endpoint on the OpenNMS instance
• The broker-url failover represents the ActiveMQ JMS server endpoint on the OpenNMS instance

Setting user credentials

You can configure credentials by using the scvcli utility in your Minion bin directory:

cd /usr/share/minion

./bin/scvcli set opennms.http <minion <user name> <minion password>
./bin/scvcli set opennms.broker <minion user name> <minion password>

Checking if the Minion connection the OpenNMS instance works

First you need to start the Minion service

service minion start

Then check if the Minion service started successfully:

● minion.service – LSB: OpenNMS Minion
   Loaded: loaded (/etc/init.d/minion; generated)
   Active: active (running) since Wed 2020-12-16 17:20:28 CET; 24s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13030 ExecStart=/etc/init.d/minion start (code=exited, status=0/SUCCESS)
    Tasks: 58 (limit: 4915)
   Cgroup: /system.slice/minion.service
           ├─13434 /bin/sh /usr/share/minion/bin/karaf server
           └─13491 /usr/lib/jvm/java-11-openjdk-armhf/bin/java -Xms128M -Xmx512M -XX:+UnlockDiagnost

Dez 16 17:20:24 openHABianDevice  ystem[1]: Starting LSB: OpenNMS Minion...
Dez 16 17:20:28 openHABianDevice  ystem[1]: Started LSB: OpenNMS Minion.

When going to your OpenNMS instance to [Admin]->[Manage Minions], you will see that the Minion successfully registered to your instance.

Same on the [Admin]->[Manage Monitoring Locations] side, here you see that the Location has been successfully generated:

Configuring Remote Monitoring

There are a bunch of activities to be performed in order to get meaningful remote monitoring information of our Natrix Gateway done to be displayed and to be getting processed on the openNMS server.

In order to fully embrace the huge capabilities openNMS gives us in terms of managing our Natrix Gateway from remote, it is vital to understand all the configuration steps which needs to be done in openNMS. First step towards this understanding is to get an overview about all the terms being used in this context.

Nodes

A node is a representation of a monitored device in openNMS.

Monitor

A Monitor is used to test a service or a specific status on a node through SNMP. The secret behind a valuable Monitoring solution is to design effective tests. A Monitor describes a certain test scenario for a specific node or even complex applications. openNMS comes with a bunch of preconfigured Monitors such as for example simple ICMP tests, complex Mail transports tests or the capability to  create own scripts for very customized tests.

Provisioning – a first clarification on the term and its co-workers

Provisioning is a mechanism to import node and service definitions and is a very powerful tool in openNMS. There are a couple of different way how to add nodes into openNMS. openNMS provides two discovery options and one import functionality.

Auto Discovery

Auto discovery is the process of automatically adding nodes to Horizon. This is also the preferred way to discover nodes which are accessible in the network the Natrix-Gateway is located in. After an Auto Discovery scan has been completed, we will see our Natrix-Gateway with all interfaces which exposed to SNMP (The Agent Behaviour configuration section). The Auto Discovery Mode is executed via the Discovery daemon and relies on the ICMP protocol.

Directed Discovery

In turn, Directed discovery is mostly used when the inventory is known, and is appropriate especially in smaller networks (up to 200 nodes). The Directed discovery mode is done through an import requisition.

Asset import

A third way to get nodes into openNMS is to import asset information from external systems through a RESTApi. A way which won’t further investigate in here.

Provisioning – how to configure

An important fact to mention is that the discovery phase as just is just about to detect certain IP addresses in a given network, nothing beyond. Provisioning in turn goes beyond and above.

The Provisioning daemon (Provisiond) is the process responsible for adding (aka provisioning) nodes, interfaces, and services into OpenNMS. If we go with the Directed Discovery, the Provisiond does provision resources from requisitions. By leveraging the Auto Discovery mode, new interfaces are discovered by scanning (via ICMP).

To detect services and applications running on the node which are worth to be monitored, the Provisiond leverages the concept of Requisitions.

Requisitions – how they work

Requisitions don’t manage nodes, but the way how services and applications can be identified on a specific node respectively set of nodes. This is done via Detectors and Policies.

openNMS comes with a default Provisioning Requisition but also support customized Provisioning Requisitions, aka user defined Provisioning Requisitions. Those user defined Provisioning Requisitions make sense to run different, customized device provisioning processes, all tailored to the specific needs for e.g., customers, business units or device types of a specific organizational unit.

Detectors

Detectors are used by Provisiond to test and probe for particular protocols and services on the discovered node. openNMS comes with many prebuild detectors which are designed to discover a certain protocol or service. Just to name some prominent one:

simple.ImapDetectorsimple.Pop3Detectorsimple.SmtpDetectorsimple.TcpDetectorsnmp.DiskUsageDetectorsnmp.SnmpDetectorssh.SshDetector

DnsDetectorDhcpDetectoricmp.IcmpDetectorjdbc.JdbcDetectorjmx.JMXDetectorsimple.FtpDetectorsimple.HttpDetectorsimple.HttpsDetector

Policies

Policies are basically rules which will be applied to discovered nodes. For example, with the MatchingIpInterfacePolicy we are able to specify which IP interfaces shall be included or excluded through a discovery scan. Similar policies exist for SNMP interfaces and node categories.

Configuration of Detectors and Policies in a Provisioning Requisition

We go to [Admin]->[ Manage Provisioning Requisitions] and add a new Requisition.

We think about a meaningful name for this Requisition (in our case we name it ‘Natrix-Project’) and start with defining the Detectors which we would like to define for this Requisition. Remember, the Detectors are there to deliver meaningful information about our node.

For the sake of usability, we just go for a few of the possible Detectors. We can always adapt them going forward.

We select [Edit Requisition] and select SNMP, SSH and the JMX-Minion. We come to the configuration of a JMX service in a later chapter.

SNMP scanning

openNMS embraces its full potential if it leverages all capabilities provided by the SNMP service running on the node. The SNMP agent, which is running on the node, is scanned for additional IP addresses, physical interfaces, and other information relating to the node. This requires the installation of a SNMP agent on our Natrix Gateway (or any other node where we want to leverage SNMP to further enrich the information which we can get from the node), along with a proper configuration of this agent. (Setting up an SNMP Agent on the Natrix Controller).

We changed the SNMP community string and the views on various different resources (=MIBS) (The Access Control configuration section). Hence, we need to change the community string in openNMS as well. We do this in [Admin]->[ Configure SNMP Community Names by IP Address].

Here we provide the IP address of (one of) the IP interfaces exposed to SNMP scans by our Natrix-Gateway (The Agent Behaviour configuration section) . While private IP addresses could appear multiple times in openNMS (in case we monitor multiple Natrix Gateways in different locations), we should also parametrize the location where our Natrix Gateway is located in.

If we press [Look up], we can adapt the RO- and RW Community strings to the value we configured in the ‘/etc/snmp/snmpd.conf’ file on our Natrix Gateway. (The Access Control configuration section)