Exception in LatencyStoringServiceMonitorAdaptor: IncorrectResultSizeDataAccessException

Description

Threshold exception after
Upgrading from older version 1.8 I got a lot of the following exception

2016-09-23 15:39:03,942 ERROR [Poller-Thread-33-of-50] o.o.n.p.p.LatencyStoringServiceMonitorAdaptor: Failed to threshold on PollableService [PollableInterface [PollableNode [3942]:172.19.40.222]:ICMP] for icmp because of an exception
org.springframework.dao.IncorrectResultSizeDataAccessException: query did not return a unique result: 2; nested exception is org.hibernate.NonUniqueResultException: query did not return a unique result: 2
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:674) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:414) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:341) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.findUnique(AbstractDaoHibernate.java:203) ~[opennms-dao-17.1.1.jar:?]
at org.opennms.netmgt.dao.hibernate.SnmpInterfaceDaoHibernate.findByNodeIdAndDescription(SnmpInterfaceDaoHibernate.java:70) ~[opennms-dao-17.1.1.jar:?]
at org.opennms.netmgt.dao.hibernate.IfLabelDaoImpl.getInterfaceInfoFromIfLabel(IfLabelDaoImpl.java:112) ~[opennms-dao-17.1.1.jar:?]
at org.opennms.netmgt.threshd.CollectionResourceWrapper.<init>(CollectionResourceWrapper.java:175) ~[opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.threshd.LatencyThresholdingSet.applyThresholds(LatencyThresholdingSet.java:105) ~[opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.applyThresholds(LatencyStoringServiceMonitorAdaptor.java:176) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.storeResponseTime(LatencyStoringServiceMonitorAdaptor.java:143) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.poll(LatencyStoringServiceMonitorAdaptor.java:118) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:119) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:191) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:293) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-17.1.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:224) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-17.1.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableNode$3.run(PollableNode.java:331) [opennms-services-17.1.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableNode.doPoll(PollableNode.java:334) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.doPoll(PollableElement.java:184) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService.doPoll(PollableService.java:226) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService$PollRunner.run(PollableService.java:63) [opennms-services-17.1.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService.doRun(PollableService.java:413) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService.run(PollableService.java:388) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.scheduler.Schedule.run(Schedule.java:142) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.scheduler.Schedule$ScheduleEntry.run(Schedule.java:86) [opennms-services-17.1.1.jar:?]
at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:209) [opennms-services-17.1.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124) [opennms-util-17.1.1.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 2
at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:899) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate$3.doInHibernate(AbstractDaoHibernate.java:198) ~[opennms-dao-17.1.1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
... 46 more

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Antonio Russo January 5, 2017 at 12:27 PM

Merged into Foundation-2016

Antonio Russo December 29, 2016 at 8:12 AM

commit 4b3ffa4d4120f573d2c05ce931ccba746da1addd
Author: Antonio <rssntn67@yahoo.it>
Date: Thu Dec 29 14:08:06 2016 +0100

https://opennms.atlassian.net/browse/NMS-8756#icft=NMS-8756: Exception in LatencyStoringServiceMonitorAdaptor

Changed the Hibernate Query to use
equals instead of LIKE

Created public branch:
MacBook-Pro-di-Antonio-Russo:opennms antonio$ git push -u origin jira/https://opennms.atlassian.net/browse/NMS-8756#icft=NMS-8756:jira/https://opennms.atlassian.net/browse/NMS-8756#icft=NMS-8756
Counting objects: 117, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (12/12), 863 bytes | 0 bytes/s, done.
Total 12 (delta 5), reused 0 (delta 0)
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To git@github.com:OpenNMS/opennms.git

And Finally crewtED THE PULL REQUEST

https://github.com/OpenNMS/opennms/pull/1217

Antonio Russo December 29, 2016 at 5:01 AM

I have set to DEBUG and found the following data...

2016-12-29 10:55:10,425 DEBUG [Poller-Thread-33-of-50] o.o.n.d.h.AbstractDaoHibernate: criteria = Criteria [class=class org.opennms.netmgt.model.OnmsSnmpInterface,aliases=[Alias [associationPat
h=node, alias=node, type=LEFT_JOIN, joinCondition=null], Alias [associationPath=ipInterfaces, alias=ipInterfaces, type=LEFT_JOIN, joinCondition=null]],restrictions=[EqRestriction [attribute=nod
e.id, value=23], NotRestriction [restriction=EqRestriction [attribute=ipInterfaces.isManaged, value=D]], EqRestriction [attribute=ipInterfaces.ipAddress, value=/10.125.50.11]],distinct=false]
2016-12-29 10:55:10,428 DEBUG [Poller-Thread-33-of-50] o.o.c.u.AlphaNumeric: parseAndReplace: original='Fa0/0' new='Fa0_0'
2016-12-29 10:55:10,428 DEBUG [Poller-Thread-33-of-50] o.o.n.d.h.IfLabelDaoImpl: getInterfaceInfoFromIfLabel: desc=Fa0_0 mac=000e83f612f4
2016-12-29 10:55:10,429 ERROR [Poller-Thread-33-of-50] o.o.n.p.p.LatencyStoringServiceMonitorAdaptor: Failed to threshold on PollableService [PollableInterface [PollableNode [23]:10.125.50.11]:
ICMP] for icmp because of an exception
org.springframework.dao.IncorrectResultSizeDataAccessException: query did not return a unique result: 3; nested exception is org.hibernate.NonUniqueResultException: query did not return a unique result: 3
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:674) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:414) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:341) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.findUnique(AbstractDaoHibernate.java:204) ~[opennms-dao-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.dao.hibernate.SnmpInterfaceDaoHibernate.findByNodeIdAndDescription(SnmpInterfaceDaoHibernate.java:70) ~[opennms-dao-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.dao.hibernate.IfLabelDaoImpl.getInterfaceInfoFromIfLabel(IfLabelDaoImpl.java:112) ~[opennms-dao-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.threshd.CollectionResourceWrapper.<init>(CollectionResourceWrapper.java:175) ~[opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.threshd.LatencyThresholdingSet.applyThresholds(LatencyThresholdingSet.java:105) ~[opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.applyThresholds(LatencyStoringServiceMonitorAdaptor.java:176) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.storeResponseTime(LatencyStoringServiceMonitorAdaptor.java:143) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.LatencyStoringServiceMonitorAdaptor.poll(LatencyStoringServiceMonitorAdaptor.java:118) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:119) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:191) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:293) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_66]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:224) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_66]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-18.0.3-SNAPSHOT.jar:?]
at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-18.0.3-SNAPSHOT.jar:?]

Antonio Russo December 29, 2016 at 4:25 AM

opennms=# select nodeid,snmpifname,snmpifdescr from snmpinterface where nodeid = 3259;
nodeid | snmpifname | snmpifdescr
------------------------------------------------------------ 3259 | Fa0/0.400 | FastEthernet0/0.400-802.1Q vLAN subif
3259 | Nu0 | Null0
3259 | Fa0/0 | FastEthernet0/0
3259 | Fa0/0.1950 | FastEthernet0/0.1950-802.1Q vLAN subif

the query result ios then...

The query is the following...

return findUnique("SELECT DISTINCT snmpIf FROM OnmsSnmpInterface AS snmpIf WHERE snmpIf.node.id = ? AND (LOWER(snmpIf.ifDescr) LIKE LOWER OR LOWER(snmpIf.ifName) LIKE LOWER)",
nodeId,
description,
description
);

opennms=# SELECT DISTINCT nodeid,snmpifname,snmpifdescr FROM snmpInterface AS snmpIf WHERE nodeid = 3259 AND (LOWER(snmpIf.snmpifDescr) LIKE LOWER('%Fa0_0%') OR LOWER(snmpIf.snmpifName) LIKE LOWER('%Fa0_0%'))
;
nodeid | snmpifname | snmpifdescr
------------------------------------------------------------ 3259 | Fa0/0 | FastEthernet0/0
3259 | Fa0/0.1950 | FastEthernet0/0.1950-802.1Q vLAN subif
3259 | Fa0/0.400 | FastEthernet0/0.400-802.1Q vLAN subif

The point is why using LIKE?

Antonio Russo December 29, 2016 at 3:48 AM

The problem is related to the fact that there are nodes with different interfaces having the same interface description.

Fixed

Details

Assignee

Reporter

Affects versions

Priority

PagerDuty

Created September 23, 2016 at 12:17 PM
Updated October 19, 2017 at 3:34 PM
Resolved January 5, 2017 at 12:27 PM