Monitoring big file system using hrStorageTable with Net-SNMP breaks Collectd

All the columns of the hrStorageTable (from HOST-RESOURCES-MIB) are 32-bits integers.

In some version of the Net-SNMP, the agent is going to show wrong information for the file systems (wrong from the point of view of the MIB, not the actual source).

Here is an example of a full walk of the hrStorageTable:

HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageIndex.7 = INTEGER: 7
HOST-RESOURCES-MIB::hrStorageIndex.10 = INTEGER: 10
HOST-RESOURCES-MIB::hrStorageIndex.31 = INTEGER: 31
HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32
HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrStorageOther
HOST-RESOURCES-MIB::hrStorageType.10 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.31 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
HOST-RESOURCES-MIB::hrStorageDescr.31 = STRING: /
HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: /sys/fs/fuse/connections
HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /media/backup
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.7 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.10 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 24728912
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 47965512
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 24728912
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 12682168
HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 23236600
HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 135373520
HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 2899736344
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 23945500
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 23945500
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 413648
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 12682168
HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 9289261
HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 2543399376

Let's take a closer look at instance 33:

HOST-RESOURCES-MIB::hrStorageIndex.33 = INTEGER: 33
HOST-RESOURCES-MIB::hrStorageType.33 = OID: HOST-RESOURCES-TYPES::hrStorageNetworkDisk
HOST-RESOURCES-MIB::hrStorageDescr.33 = STRING: /media/backup
HOST-RESOURCES-MIB::hrStorageAllocationUnits.33 = INTEGER: 1024 Bytes
HOST-RESOURCES-MIB::hrStorageSize.33 = INTEGER: 2899736344
HOST-RESOURCES-MIB::hrStorageUsed.33 = INTEGER: 2543399376

The size in bytes for hrStorageSize is hrStorageSize * hrStorageAllocationUnits, in this case something like 2.5 Tb.

The number displayed on hrStorageSize and hrStorageUsed cannot be assigned to a 32-bits signed integer, and that is the problem.

For this reason, SNMP4J is going to complain about it because the numbers are not correct, and the following exception is generated:

3057 [Timer-0] DEBUG org.snmp4j.transport.DefaultUdpTransportMapping - Sending message to 192.168.210.11/161 with length 47: 30:2d:02:01:01:04:07:62:69:6e:6b:73:65:63:a1:1f:02:04:5a:db:fa:3a:02:01:00:02:01:00:30:11:30:0f:06:0b:2b:06:01:02:01:19:02:03:01:05:20:05:00
3078 [DefaultUDPTransportMapping_127.0.0.1/0] DEBUG org.snmp4j.transport.DefaultUdpTransportMapping - Received message from /192.168.210.11/161 with length 52: 30:32:02:01:01:04:07:62:69:6e:6b:73:65:63:a2:24:02:04:5a:db:fa:3a:02:01:00:02:01:00:30:16:30:14:06:0b:2b:06:01:02:01:19:02:03:01:05:21:02:05:00:ac:d6:77:18
java.io.IOException: Length greater than 32bit are not supported for integers: at position 47
at org.snmp4j.asn1.BER.decodeInteger(BER.java:626)
at org.snmp4j.smi.Integer32.decodeBER(Integer32.java:60)
at org.snmp4j.smi.AbstractVariable.createFromBER(AbstractVariable.java:172)
at org.snmp4j.smi.VariableBinding.decodeBER(VariableBinding.java:167)
at org.snmp4j.PDU.decodeBER(PDU.java:538)
at org.snmp4j.mp.MPv2c.prepareDataElements(MPv2c.java:195)
at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:272)
at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:368)
at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:328)
at org.snmp4j.transport.AbstractTransportMapping.fireProcessMessage(AbstractTransportMapping.java:76)
at org.snmp4j.transport.DefaultUdpTransportMapping$ListenThread.run(DefaultUdpTransportMapping.java:378)
at java.lang.Thread.run(Thread.java:662)
3078 [DefaultUDPTransportMapping_127.0.0.1/0] WARN org.snmp4j.MessageDispatcherImpl - java.io.IOException: Length greater than 32bit are not supported for integers: at position 47

This is basically a bug of the Net-SNMP agent implementation, but when this happen Collectd is not going to update ANY metric for this node. So hrStorageTable is breaking the SNMP data collection of the whole node.