Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8376

Poller threads can get locked up while waiting for outages to be created/resolved

    XMLWordPrintable

    Details

      Description

      The segments of the following stack trace show that several poller threads are locked, waiting to append an event to a linked list, while another thread, which has a lock on the linked list in processing the outages.

      Ideally, the poller threads should not need to block and wait for this lock.

      24 threads with this stack:
      "Poller-Thread-10-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-16-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-17-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-18-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-20-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-21-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-22-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-27-of-40": waiting to acquire [0x0000000707f46848]24
      "Poller-Thread-28-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-29-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-3-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-30-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-31-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-33-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-35-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-36-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-37-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-39-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-4-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-40-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-5-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-6-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-8-of-40": waiting to acquire [0x0000000707f46848]
      "Poller-Thread-9-of-40": waiting to acquire [0x0000000707f46848]
      	at org.opennms.netmgt.poller.DefaultPollContext.sendEvent(DefaultPollContext.java:234)
      	at org.opennms.netmgt.poller.pollables.PollableElement.processGoingDown(PollableElement.java:407)
      
      "OpenNMS.Poller.DefaultPollContext-Thread": running, holding [0x0000000729a94c98, 0x0000000729bd5bc0, 0x0000000707f46848]
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.read(SocketInputStream.java:152)
      	at java.net.SocketInputStream.read(SocketInputStream.java:122)
      	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
      	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
      	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
      	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
      	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1704)
      	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
      	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
      	at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)
      	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2591)
      	at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495)
      	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822)
      	at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
      	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
      	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
      	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
      	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
      	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
      	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
      	at org.springframework.orm.hibernate3.HibernateAccessor.flushIfNecessary(HibernateAccessor.java:390)
      	at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:408)
      	at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:375)
      	at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:738)
      	at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.saveOrUpdate(AbstractDaoHibernate.java:410)
      	at org.opennms.netmgt.poller.QueryManagerDaoImpl.updateResolvedOutageWithEventId(QueryManagerDaoImpl.java:175)
      	at org.opennms.netmgt.poller.DefaultPollContext$2.run(DefaultPollContext.java:351)
      	at org.opennms.netmgt.poller.pollables.PendingPollEvent.processPending(PendingPollEvent.java:145)
      	at org.opennms.netmgt.poller.DefaultPollContext.onEvent(DefaultPollContext.java:408)
      	at org.opennms.netmgt.eventd.EventIpcManagerDefaultImpl$EventListenerExecutor$2.run(EventIpcManagerDefaultImpl.java:176)
      

        Attachments

          Activity

            People

            • Assignee:
              seth Seth Leger
              Reporter:
              j-white Jesse White
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: