Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 16.0.3
-
Component/s: Database
-
Security Level: Default (Default Security Scheme)
-
Labels:None
-
Sprint:Newts - Phase 6, Newts - Sprint 7
Description
Seeing a ton of deadlocks on Demo, across multiple daemons. Not sure if this is a problem with the database, or OpenNMS, but it needs investigation.
2015-09-15 11:48:52,839 ERROR [EventIpcManagerDefaultImpl-Thread-2-of-5] o.h.u.JDBCExceptionReporter: ERROR: deadlock detected Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953. Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969. Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944. Hint: See server log for query details. 2015-09-15 11:48:52,846 WARN [EventIpcManagerDefaultImpl-Thread-2-of-5] o.o.n.e.DefaultEventHandlerImpl: Unknown exception processing event with processor org.opennms.netmgt.eventd.processor.HibernateEventWriter@1efb33c5; not processing with any later processors. org.springframework.dao.DeadlockLoserDataAccessException: Hibernate operation: could not update: [org.opennms.netmgt.model.OnmsAssetRecord#45]; SQL [update assets set additionalhardware=?, admin=?, assetNumber=?, autoenable=?, building=?, category=?, circuitId=?, comment=?, connection=?, cpu=?, dateInstalled=?, department=?, description=?, displayCategory=?, division=?, enable=?, floor=?, address1=?, address2=?, city=?, country=?, latitude=?, longitude=?, state=?, zip=?, hdd1=?, hdd2=?, hdd3=?, hdd4=?, hdd5=?, hdd6=?, inputpower=?, userLastModified=?, lastModifiedDate=?, lease=?, leaseExpires=?, maintContractExpires=?, maintcontract=?, managedObjectInstance=?, managedObjectType=?, manufacturer=?, modelNumber=?, nodeId=?, notifyCategory=?, numpowersupplies=?, operatingSystem=?, password=?, pollerCategory=?, port=?, rack=?, rackunitheight=?, ram=?, region=?, room=?, serialNumber=?, slot=?, snmpcommunity=?, storagectrl=?, supportPhone=?, thresholdCategory=?, username=?, vendor=?, vendorAssetNumber=?, vendorFax=?, vendorPhone=?, vmwareManagedEntityType=?, vmwareManagedObjectId=?, vmwareManagementServer=?, vmwareState=?, vmwareTopologyInfo=? where id=?]; ERROR: deadlock detected Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953. Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969. Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944. Hint: See server log for query details.; nested exception is org.postgresql.util.PSQLException: ERROR: deadlock detected Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953. Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969. Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944. Hint: See server log for query details. at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:263) ~[org.apache.servicemix.bundles.spring-jdbc-4.0.5.RELEASE_1.jar:?] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[org.apache.servicemix.bundles.spring-jdbc-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:426) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:412) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:379) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:519) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:512) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.get(AbstractDaoHibernate.java:370) ~[opennms-dao-16.0.3.jar:?] at org.opennms.netmgt.eventd.processor.HibernateEventWriter.insertEvent(HibernateEventWriter.java:170) ~[org.opennms.features.events.daemon-16.0.3.jar:?] at org.opennms.netmgt.eventd.processor.HibernateEventWriter.process(HibernateEventWriter.java:145) ~[org.opennms.features.events.daemon-16.0.3.jar:?] at org.opennms.netmgt.eventd.DefaultEventHandlerImpl$EventHandlerRunnable.run(DefaultEventHandlerImpl.java:135) [org.opennms.features.events.daemon-16.0.3.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_40] at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124) [opennms-util-16.0.3.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953. Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969. Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944. Hint: See server log for query details. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) ~[postgresql-9.3-1100-jdbc4.jar:?] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) ~[postgresql-9.3-1100-jdbc4.jar:?] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) ~[postgresql-9.3-1100-jdbc4.jar:?] at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560) ~[postgresql-9.3-1100-jdbc4.jar:?] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) ~[postgresql-9.3-1100-jdbc4.jar:?] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363) ~[postgresql-9.3-1100-jdbc4.jar:?] at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) ~[c3p0-0.9.1.2.jar:0.9.1.2] at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2591) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final] at org.springframework.orm.hibernate3.HibernateAccessor.flushIfNecessary(HibernateAccessor.java:392) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:412) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?] ... 11 more
Attachments
Issue Links
- depends on
-
NMS-7755 c.m.v.a.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@59804d53 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
-
- Resolved
-