Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-5747

Monitoring big file system using hrStorageTable with Net-SNMP breaks Collectd

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.11.3, 1.10.8
    • Fix Version/s: 1.10.9, 1.11.90
    • Labels:
    • Environment:
      Initial Test: Ubuntu 8.04 LTS with Net-SNMP 5.4.2.1

      Also tested against several Solaris machines like "Sun SNMP Agent, Ultra-5_10" (.1.3.6.1.4.1.42) and "Sun SNMP Agent, Sun-Blade-1500" (.1.3.6.1.4.1.42.2.1.1)

      Description

      All the columns of the hrStorageTable (from HOST-RESOURCES-MIB) are 32-bits integers.

      In some version of the Net-SNMP, the agent is going to show wrong information for the file systems (wrong from the point of view of the MIB, not the actual source).

      Here is an example of a full walk of the hrStorageTable:

      HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
      HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
      HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
      HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
      HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
      HOST-RESOURCES-MIB::hrStorageIndex.31 = INTEGER: 31
      HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32
      HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
      HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRam
      HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
      HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageOther
      HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageOther
      HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
      HOST-RESOURCES-MIB::hrStorageType.31 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
      HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
      HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
      HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
      HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
      HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
      HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
      HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
      HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
      HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /sys/fs/fuse/connections
      HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /media/backup
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 24728912
      HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 47965512
      HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 24728912
      HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 12682168
      HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 23236600
      HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 135373520
      HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 0
      HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 2899736344
      HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 23945500
      HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 23945500
      HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 413648
      HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 12682168
      HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 0
      HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 9289261
      HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 0
      HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 2543399376

      Let's take a closer look at instance 33:

      HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
      HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
      HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /media/backup
      HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 1024 Bytes
      HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 2899736344
      HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 2543399376

      The size in bytes for hrStorageSize is hrStorageSize * hrStorageAllocationUnits, in this case something like 2.5 Tb.

      The number displayed on hrStorageSize and hrStorageUsed cannot be assigned to a 32-bits signed integer, and that is the problem.

      For this reason, SNMP4J is going to complain about it because the numbers are not correct, and the following exception is generated:

      3057 [Timer-0] DEBUG org.snmp4j.transport.DefaultUdpTransportMapping - Sending message to 192.168.210.11/161 with length 47: 30:2d:02:01:01:04:07:62:69:6e:6b:73:65:63:a1:1f:02:04:5a:db:fa:3a:02:01:00:02:01:00:30:11:30:0f:06:0b:2b:06:01:02:01:19:02:03:01:05:20:05:00
      3078 [DefaultUDPTransportMapping_127.0.0.1/0] DEBUG org.snmp4j.transport.DefaultUdpTransportMapping - Received message from /192.168.210.11/161 with length 52: 30:32:02:01:01:04:07:62:69:6e:6b:73:65:63:a2:24:02:04:5a:db:fa:3a:02:01:00:02:01:00:30:16:30:14:06:0b:2b:06:01:02:01:19:02:03:01:05:21:02:05:00:ac:d6:77:18
      java.io.IOException: Length greater than 32bit are not supported for integers: at position 47
      at org.snmp4j.asn1.BER.decodeInteger(BER.java:626)
      at org.snmp4j.smi.Integer32.decodeBER(Integer32.java:60)
      at org.snmp4j.smi.AbstractVariable.createFromBER(AbstractVariable.java:172)
      at org.snmp4j.smi.VariableBinding.decodeBER(VariableBinding.java:167)
      at org.snmp4j.PDU.decodeBER(PDU.java:538)
      at org.snmp4j.mp.MPv2c.prepareDataElements(MPv2c.java:195)
      at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:272)
      at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:368)
      at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:328)
      at org.snmp4j.transport.AbstractTransportMapping.fireProcessMessage(AbstractTransportMapping.java:76)
      at org.snmp4j.transport.DefaultUdpTransportMapping$ListenThread.run(DefaultUdpTransportMapping.java:378)
      at java.lang.Thread.run(Thread.java:662)
      3078 [DefaultUDPTransportMapping_127.0.0.1/0] WARN org.snmp4j.MessageDispatcherImpl - java.io.IOException: Length greater than 32bit are not supported for integers: at position 47

      This is basically a bug of the Net-SNMP agent implementation, but when this happen Collectd is not going to update ANY metric for this node. So hrStorageTable is breaking the SNMP data collection of the whole node.

        Attachments

          Activity

            People

            • Assignee:
              agalue Alejandro Galue
              Reporter:
              agalue Alejandro Galue
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: