Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-9381

Erroneous outages/failures triggered when stopping OpenNMS

    XMLWordPrintable

    Details

    • Sprint:
      Horizon - May 24th, Horizon - May 31st

      Description

      Similar to NMS-8975 and NMS-8976, we have discovered more cases where pollerd can trigger false outages due to unexpected exceptions. collectd suffers from similar behavior too.

      java.lang.InterruptedException

      2017-05-25 14:39:52,179 ERROR [Poller-Thread-74-of-300] o.o.n.p.p.PollableServiceConfig: Unexpected exception while polling PollableService[location=CTC-KC-II, interface=PollableInterface [PollableNode [576]:7.4.140.225], svcName=ICMP]. Marking service as DOWN
      java.util.concurrent.ExecutionException: java.lang.InterruptedException
              at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_131]
              at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:133) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:190) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:293) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableNode$3.run(PollableNode.java:331) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableNode.doPoll(PollableNode.java:334) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.doPoll(PollableElement.java:184) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.doPoll(PollableService.java:214) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService$PollRunner.run(PollableService.java:60) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.doRun(PollableService.java:404) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.run(PollableService.java:379) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.scheduler.Schedule.run(Schedule.java:142) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.scheduler.Schedule$ScheduleEntry.run(Schedule.java:86) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
              at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124) [opennms-util-19.1.1-SNAPSHOT.jar:?]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
      Caused by: java.lang.InterruptedException
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_131]
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_131]
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_131]
              at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:90) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:383) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:351) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:241) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:351) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:209) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:128) ~[camel-core-2.14.4.jar:2.14.4]
              at org.apache.camel.impl.DefaultProducerTemplate$15.call(DefaultProducerTemplate.java:636) ~[camel-core-2.14.4.jar:2.14.4]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
              ... 1 more
      
      

      java.lang.RejectedExecutionException

      2017-05-25 14:26:58,070 ERROR [Poller-Thread-5-of-30] o.o.n.p.p.PollableServiceConfig: Unexpected exception while polling PollableService[location=CTC-KC-II, interface=PollableInterface [PollableNode [552]:7.4.150.142], svcName=HTTP]. Marking service as DOWN
      java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
              at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_131]
              at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:133) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:190) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:293) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableNode$3.run(PollableNode.java:331) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:228) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableNode.doPoll(PollableNode.java:334) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.doPoll(PollableElement.java:184) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.doPoll(PollableService.java:214) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService$PollRunner.run(PollableService.java:60) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:264) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:250) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.doRun(PollableService.java:404) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.poller.pollables.PollableService.run(PollableService.java:379) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.scheduler.Schedule.run(Schedule.java:142) [opennms-services-19.1.1-SNAPSHOT.jar:?]
              at org.opennms.netmgt.scheduler.Schedule$ScheduleEntry.run(Schedule.java:86) [opennms-services-19.1.1-SNAPSHOT.jar:?]
      

        Attachments

          Activity

            People

            • Assignee:
              j-white Jesse White
              Reporter:
              j-white Jesse White
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: