SNMP collection failing for "interface label is null or blank"

Description

Seeing this error in collectd.log:

 

2018-07-17 07:09:15,022 ERROR [Collectd-Thread-28-of-50] o.o.n.c.CollectableService: An undeclared throwable was caught during data collection for interface 1415/192.168.30.4/SNMP

Caused by: java.lang.IllegalStateException: Could not construct resource directory because interface label is null or blank: nodeId: 1415

ifTable for above device is:

 

SNMP table: IF-MIB::ifTable

ifIndex ifDescr ifType ifMtu ifSpeed ifPhysAddress ifAdminStatus ifOperStatus ifLastChange ifInOctets ifInUcastPkts ifInNUcastPkts ifInDiscards ifInErrors ifInUnknownProtos ifOutOctets ifOutUcastPkts ifOutNUcastPkts ifOutDiscards ifOutErrors ifOutQLen ifSpecific
1 "Ethernet1" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E B7 " up up 0:0:14:03.29 641477203 2582142406 3873249296 398145 0 0 1591041779 1500327419 573959260 0 0 0 EtherLike-MIB::dot3
2 "Ethernet2" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E B8 " up up 0:0:14:04.79 2963103320 3105852021 3852320696 161274 1 0 2288352283 2415411141 554805060 0 0 0 EtherLike-MIB::dot3
3 "Ethernet3" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E B9 " up up 90:18:04:52.12 3597944789 367165879 258509 1292 0 0 3469022058 131198075 51735701 0 2 0 EtherLike-MIB::dot3
4 "Ethernet4" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BA " up up 0:0:46:06.62 3913512691 112826 2074608444 2 0 0 4284897588 129010 36233565 0 0 0 EtherLike-MIB::dot3
5 "Ethernet5" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BB " up up 90:18:05:05.91 3748923115 2121190505 141261347 0 0 0 2851572744 2250260983 741670087 0 1 0 EtherLike-MIB::dot3
6 "Ethernet6" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BC " up up 39:23:47:24.92 2241232338 109586315 861886114 624 0 0 3152688813 133294110 3725851858 0 2 0 EtherLike-MIB::dot3
7 "Ethernet7" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BD " up up 0:0:38:03.54 1352036614 44383603 41038921 84666 0 0 344687608 946457710 177649255 0 1 0 EtherLike-MIB::dot3
8 "Ethernet8" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BE " up up 63:1:38:27.05 1361784745 1710881488 236536 61688 0 0 1423673604 2807588760 62766657 0 2 0 EtherLike-MIB::dot3
9 "Ethernet9" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E BF " up up 0:0:50:32.57 1107930652 359473241 3895347816 94 0 0 388533064 7083912 3620208324 0 0 0 EtherLike-MIB::dot3
10 "Ethernet10" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C0 " up up 90:18:08:18.51 3467946679 462207165 1779178604 2 1 0 2686115858 535512337 1466249811 0 1 0 EtherLike-MIB::dot3
11 "Ethernet11" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C1 " up up 0:0:47:26.78 3349209724 451949368 1774189060 0 0 0 1295865438 517533963 1402439075 0 0 0 EtherLike-MIB::dot3
12 "Ethernet12" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C2 " up up 0:0:51:17.67 297166993 2257761655 235926 337737 0 0 1050592912 2371584411 50300693 0 0 0 EtherLike-MIB::dot3
13 "Ethernet13" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C3 " up up 0:0:51:18.68 488878891 2230898331 136383316 35 0 0 2921543792 1716974830 3476687294 0 0 0 EtherLike-MIB::dot3
14 "Ethernet14" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C4 " up up 0:0:50:30.56 1058331664 60002869 1757569104 246 0 0 241959433 73968588 2557577522 0 0 0 EtherLike-MIB::dot3
15 "Ethernet15" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C5 " up up 0:0:49:11.20 3932499305 405294131 3948640141 0 0 0 280376869 466243670 4197173267 0 0 0 EtherLike-MIB::dot3
16 "Ethernet16" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C6 " up up 0:0:48:46.91 3302068512 113731246 2043083338 481 0 0 706368796 137761559 772282380 0 0 0 EtherLike-MIB::dot3
17 "Ethernet17" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C7 " up up 0:0:02:54.88 714531672 877680700 398985909 186225 0 0 2712259183 2596557938 2036369237 0 0 0 EtherLike-MIB::dot3
18 "Ethernet18" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C8 " up up 0:0:01:15.97 970797240 168796341 1955454433 266827 0 0 4176712755 3736611325 4136061419 0 0 0 EtherLike-MIB::dot3
19 "Ethernet19" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E C9 " up up 0:0:01:16.72 3311229793 2755327036 2732449931 286029 0 0 873271554 2462697295 3118225515 0 0 0 EtherLike-MIB::dot3
20 "Ethernet20" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E CA " up up 0:0:01:11.23 3281008998 1548947841 2706618551 680886 0 0 159791885 3754324644 1369243193 0 0 0 EtherLike-MIB::dot3
21 "Ethernet21" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E CB " up up 0:0:01:08.95 2871927044 3868365699 2756798809 367551 0 0 3075054502 2992618381 648035033 0 0 0 EtherLike-MIB::dot3
22 "Ethernet22" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E CC " up up 132:22:57:26.70 3326776387 2764353438 64612809 587919 0 0 1473500599 2570667306 3659501090 0 2 0 EtherLike-MIB::dot3
23 "Ethernet23" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E CD " up 6 133:0:09:55.73 2403046280 2282203419 56322843 741653 46 0 1859336130 3281057737 3115823353 0 0 0 EtherLike-MIB::dot3
24 "Ethernet24" ethernet-csmacd 9214 4294967295 "00 1C 73 13 2E CE " up up 0:0:02:54.87 662405242 782486055 115843296 126279 0 0 2459104657 2710659873 1910049392 0 0 0 EtherLike-MIB::dot3
999001 "Management1" ethernet-csmacd 1500 10000000 "00 1C 73 13 2E B5 " up down 0:0:00:58.59 0 0 0 0 0 0 0 0 0 0 0 0 EtherLike-MIB::dot3
1000001 "Port-Channel1" 161 9214 0 "00 00 00 00 00 00 " up 7 132:23:24:20.07 1394645143 751571075 120798257 1329572 46 0 763952039 1556508485 2431671339 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
1000002 "Port-Channel2" 161 9214 4294967295 "00 1C 73 13 2E C7 " up up 0:0:03:00.80 954813033 1659850037 514828968 312504 0 0 831077343 1012218111 3946418607 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
1000003 "Port-Channel3" 161 9214 4294967295 "00 1C 73 13 2E C8 " up up 0:0:01:18.98 4277698233 2924122562 392921549 552856 0 0 755016281 1904341324 2959319633 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
1000004 "Port-Channel4" 161 9214 4294967295 "00 1C 73 13 2E CA " up up 0:0:01:11.07 1851533375 1122342759 1168435871 1048437 0 0 3234846131 2451975729 2017278224 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
1000005 "Port-Channel5" 161 9214 4294967295 "00 1C 73 13 2E B7 " up up 0:0:14:06.82 3593751571 1392990628 3430601790 559419 1 0 3867875664 3915706746 1128764298 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
1000011 "Port-Channel11" 161 9214 4294967295 "00 1C 73 13 2E CC " up up 132:22:57:26.71 26885184 9393 137388 0 0 0 2568881818 249252 48685780 0 0 0 IEEE8023-LAG-MIB::dot3adAgg
2000001 "Vlan1" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:08.99 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
2000042 "Vlan42" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:11.23 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
2000080 "Vlan80" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:11.23 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
2000131 "Vlan131" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:11.23 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
2000142 "Vlan142" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:11.23 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
2000180 "Vlan180" 136 1500 0 "00 1C 73 13 2E B6 " up up 0:0:01:11.23 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero
5000175 "Loopback175" softwareLoopback 65535 0 "" up up 0:0:00:00.00 0 0 0 0 0 0 0 0 0 0 0 0 SNMPv2-SMI::zeroDotZero

 

snmpinterfaces table for device:

 ----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

id

nodeid

snmpphysaddr

snmpifindex

snmpifdescr

snmpiftype

snmpifname

snmpifspeed

snmpifadminstatus

snmpifoperstatus

snmpifalias

snmpcollect

snmplastcapsdpoll

snmppoll

snmplastsnmppoll

hasflows

----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

57541

1415

001c73132eb7

1

Ethernet1

6

Ethernet1

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57542

1415

001c73132eb8

2

Ethernet2

6

Ethernet2

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57543

1415

001c73132eb9

3

Ethernet3

6

Ethernet3

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57544

1415

001c73132eba

4

Ethernet4

6

Ethernet4

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

57545

1415

001c73132ebb

5

Ethernet5

6

Ethernet5

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

57546

1415

001c73132ebc

6

Ethernet6

6

Ethernet6

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57547

1415

001c73132ebd

7

Ethernet7

6

Ethernet7

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

57548

1415

001c73132ebe

8

Ethernet8

6

Ethernet8

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57549

1415

001c73132ebf

9

Ethernet9

6

Ethernet9

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57550

1415

001c73132ec0

10

Ethernet10

6

Ethernet10

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57551

1415

001c73132ec1

11

Ethernet11

6

Ethernet11

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57552

1415

001c73132ec2

12

Ethernet12

6

Ethernet12

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

57553

1415

001c73132ec3

13

Ethernet13

6

Ethernet13

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57554

1415

001c73132ec4

14

Ethernet14

6

Ethernet14

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57555

1415

001c73132ec5

15

Ethernet15

6

Ethernet15

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57556

1415

001c73132ec6

16

Ethernet16

6

Ethernet16

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

93384

1415

001c73132ec7

17

Ethernet17

6

Ethernet17

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93385

1415

001c73132ec8

18

Ethernet18

6

Ethernet18

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93386

1415

001c73132ec9

19

Ethernet19

6

Ethernet19

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93387

1415

001c73132eca

20

Ethernet20

6

Ethernet20

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93388

1415

001c73132ecb

21

Ethernet21

6

Ethernet21

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93389

1415

001c73132ecc

22

Ethernet22

6

Ethernet22

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

93390

1415

001c73132ecd

23

Ethernet23

6

Ethernet23

10000000000

1

6

XXX

N

2018-08-03 15:58:04.98-05

N

NULL

f

93391

1415

001c73132ece

24

Ethernet24

6

Ethernet24

10000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

84385

1415

001c73132eb5

999001

Management1

6

Management1

10000000

1

2

 XXX

N

2018-08-03 15:58:04.98-05

N

NULL

f

93392

1415

000000000000

1000001

Port-Channel1

161

Port-Channel1

0

1

7

XXX

N

2018-08-03 15:58:04.98-05

N

NULL

f

122259

1415

001c73132ec7

1000002

Port-Channel2

161

Port-Channel2

20000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

122260

1415

001c73132ec8

1000003

Port-Channel3

161

Port-Channel3

20000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

122261

1415

001c73132eca

1000004

Port-Channel4

161

Port-Channel4

20000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

159957

1415

001c73132eb7

1000005

Port-Channel5

161

Port-Channel5

20000000000

1

1

XXX

UC

2018-08-03 15:58:04.98-05

N

NULL

f

453870

1415

001c73132ecc

1000011

Port-Channel11

161

Port-Channel11

10000000000

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

84275

1415

001c73132eb6

2000001

Vlan1

136

Vlan1

0

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

57566

1415

001c73132eb6

2000042

Vlan42

136

Vlan42

0

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

111957

1415

001c73132eb6

2000080

Vlan80

136

Vlan80

0

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

93393

1415

001c73132eb6

2000131

Vlan131

136

Vlan131

0

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

84183

1415

001c73132eb6

2000142

Vlan142

136

Vlan142

0

1

1

XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

344379

1415

001c73132eb6

2000180

Vlan180

136

Vlan180

0

1

1

 XXX

C

2018-08-03 15:58:04.98-05

N

NULL

f

476887

1415

NULL

5000175

NULL

NULL

NULL

NULL

NULL

NULL

NULL

C

NULL

NULL

NULL

f

----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

 

Attached is a screenshot of the opennms/admin/snmpInterfaces.jsp page showing what I think is the offending interface, as well as a screenshot showing the impact that this is having on SNMP datacollection.

It would appear as though ONMS is automatically adding, then removing this interface dynamically throughout the day...

 

 

Environment

RHEL7

Acceptance / Success Criteria

None

Attachments

2
  • 17 Jul 2018, 09:06 PM
  • 17 Jul 2018, 09:06 PM

Lucidchart Diagrams

Activity

Show:

Jesse White June 10, 2021 at 8:56 PM

Gordon Oliver May 23, 2019 at 9:12 PM

I'm seeing this in Horizon 24.0 - is there a work around?

Gordon Oliver May 23, 2019 at 2:24 PM

I'm seeing the same issue in Ubuntu 18.04 LTS.

Aaron Scamehorn August 3, 2018 at 9:04 PM

So, I manually delete the row from snmpinterfaces with NULL values (snmpifindex=5000175), and it eventually comes back...

Fixed

Details

Assignee

Reporter

Affects versions

Priority

PagerDuty

Created July 17, 2018 at 10:02 PM
Updated June 11, 2021 at 4:54 PM
Resolved June 11, 2021 at 4:54 PM

Flag notifications