Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 16.0.3
    • Fix Version/s: 16.0.4, 17.0.0
    • Component/s: Database
    • Security Level: Default (Default Security Scheme)
    • Labels:
      None
    • Sprint:
      Newts - Phase 6, Newts - Sprint 7

      Description

      Seeing a ton of deadlocks on Demo, across multiple daemons. Not sure if this is a problem with the database, or OpenNMS, but it needs investigation.

      2015-09-15 11:48:52,839 ERROR [EventIpcManagerDefaultImpl-Thread-2-of-5] o.h.u.JDBCExceptionReporter: ERROR: deadlock detected
        Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953.
      Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969.
      Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944.
        Hint: See server log for query details.
      2015-09-15 11:48:52,846 WARN  [EventIpcManagerDefaultImpl-Thread-2-of-5] o.o.n.e.DefaultEventHandlerImpl: Unknown exception processing event with processor org.opennms.netmgt.eventd.processor.HibernateEventWriter@1efb33c5; not processing with any later processors.
      org.springframework.dao.DeadlockLoserDataAccessException: Hibernate operation: could not update: [org.opennms.netmgt.model.OnmsAssetRecord#45]; SQL [update assets set additionalhardware=?, admin=?, assetNumber=?, autoenable=?, building=?, category=?, circuitId=?, comment=?, connection=?, cpu=?, dateInstalled=?, department=?, description=?, displayCategory=?, division=?, enable=?, floor=?, address1=?, address2=?, city=?, country=?, latitude=?, longitude=?, state=?, zip=?, hdd1=?, hdd2=?, hdd3=?, hdd4=?, hdd5=?, hdd6=?, inputpower=?, userLastModified=?, lastModifiedDate=?, lease=?, leaseExpires=?, maintContractExpires=?, maintcontract=?, managedObjectInstance=?, managedObjectType=?, manufacturer=?, modelNumber=?, nodeId=?, notifyCategory=?, numpowersupplies=?, operatingSystem=?, password=?, pollerCategory=?, port=?, rack=?, rackunitheight=?, ram=?, region=?, room=?, serialNumber=?, slot=?, snmpcommunity=?, storagectrl=?, supportPhone=?, thresholdCategory=?, username=?, vendor=?, vendorAssetNumber=?, vendorFax=?, vendorPhone=?, vmwareManagedEntityType=?, vmwareManagedObjectId=?, vmwareManagementServer=?, vmwareState=?, vmwareTopologyInfo=? where id=?]; ERROR: deadlock detected
        Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953.
      Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969.
      Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944.
        Hint: See server log for query details.; nested exception is org.postgresql.util.PSQLException: ERROR: deadlock detected
        Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953.
      Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969.
      Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944.
        Hint: See server log for query details.
              at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:263) ~[org.apache.servicemix.bundles.spring-jdbc-4.0.5.RELEASE_1.jar:?]
              at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[org.apache.servicemix.bundles.spring-jdbc-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:426) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:412) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:379) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:519) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:512) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.get(AbstractDaoHibernate.java:370) ~[opennms-dao-16.0.3.jar:?]
              at org.opennms.netmgt.eventd.processor.HibernateEventWriter.insertEvent(HibernateEventWriter.java:170) ~[org.opennms.features.events.daemon-16.0.3.jar:?]
              at org.opennms.netmgt.eventd.processor.HibernateEventWriter.process(HibernateEventWriter.java:145) ~[org.opennms.features.events.daemon-16.0.3.jar:?]
              at org.opennms.netmgt.eventd.DefaultEventHandlerImpl$EventHandlerRunnable.run(DefaultEventHandlerImpl.java:135) [org.opennms.features.events.daemon-16.0.3.jar:?]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_40]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_40]
              at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124) [opennms-util-16.0.3.jar:?]
              at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
      Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
        Detail: Process 19944 waits for ShareLock on transaction 34756332; blocked by process 19953.
      Process 19953 waits for ShareLock on transaction 34756333; blocked by process 19969.
      Process 19969 waits for ExclusiveLock on tuple (15,24) of relation 17727 of database 17667; blocked by process 19944.
        Hint: See server log for query details.
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363) ~[postgresql-9.3-1100-jdbc4.jar:?]
              at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) ~[c3p0-0.9.1.2.jar:0.9.1.2]
              at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2591) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) ~[hibernate-core-3.6.10.Final.jar:3.6.10.Final]
              at org.springframework.orm.hibernate3.HibernateAccessor.flushIfNecessary(HibernateAccessor.java:392) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:412) ~[org.apache.servicemix.bundles.spring-orm-4.0.5.RELEASE_1.jar:?]
              ... 11 more
      

        Attachments

        1. deadlock_pg_log.txt
          42 kB
        2. deadlock.txt
          260 kB
        3. postgresql-Fri.log
          6.05 MB
        4. provisiond.log
          72.00 MB
        5. th_dump_core.tar
          3.28 MB
        6. th_dump_GUI.tar
          370 kB

          Issue Links

            Activity

              People

              • Assignee:
                j-white Jesse White
                Reporter:
                ranger Benjamin Reed
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: