Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13508

Node cache gets out of sync with database

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • None
    • 30.0.0
    • None
    • Security Level: Default (Default Security Scheme)
    • 3
    • Horizon 2021 - Nov 11 - Dec 1
    • Backlog
    • 705

    Description

      pollerd is attempting to poll service on nodes that have been deleted from the database:

      2021-07-15 20:16:19,864 ERROR [Poller-Thread-418-of-500]
      o.o.n.p.QueryManagerDaoImpl: Failed to set the last good/fail
      timestamp for service named ICMP on node id 16545 and
      interface /172.30.96.4 for Down.
      java.util.NoSuchElementException:
      Service no longer exists.
             
      at
      org.opennms.netmgt.poller.QueryManagerDaoImpl.lambda$updateLastGoodOrFail$0(QueryManagerDaoImpl.java:342)
      ~[opennms-services-28.0.0.jar:?]
             
      at
      org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
      ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
             
      at
      org.opennms.netmgt.poller.QueryManagerDaoImpl.updateLastGoodOrFail(QueryManagerDaoImpl.java:338)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.DefaultPollContext.trackPoll(DefaultPollContext.java:445)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableServiceConfig.poll(PollableServiceConfig.java:142)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableService.poll(PollableService.java:191)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.poll(PollableElement.java:309)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319)
      [opennms-services-28.0.0.jar:?]
             
      at
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      [?:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:240)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:227)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableInterface.poll(PollableInterface.java:228)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableContainer$5.run(PollableContainer.java:319)
      [opennms-services-28.0.0.jar:?]
             
      at
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      [?:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:240)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:227)
      [opennms-services-28.0.0.jar:?]
          
         at
      org.opennms.netmgt.poller.pollables.PollableContainer.poll(PollableContainer.java:326)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableNode$3.run(PollableNode.java:288)
      [opennms-services-28.0.0.jar:?]
             
      at
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      [?:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:240)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:227)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableNode.doPoll(PollableNode.java:291)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.doPoll(PollableElement.java:184)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableService.doPoll(PollableService.java:215)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableService$PollRunner.run(PollableService.java:61)
      [opennms-services-28.0.0.jar:?]
             
      at
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      [?:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:276)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableElement.withTreeLock(PollableElement.java:259)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableService.doRun(PollableService.java:405)
      [opennms-services-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.poller.pollables.PollableService.run(PollableService.java:380)
      [opennms-services-28.0.0.jar:?]
             
      at org.opennms.netmgt.scheduler.Schedule.run(Schedule.java:142)
      [org.opennms.core.daemon-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.scheduler.Schedule$ScheduleEntry.run(Schedule.java:86)
      [org.opennms.core.daemon-28.0.0.jar:?]
             
      at
      org.opennms.netmgt.scheduler.LegacyScheduler$1.run(LegacyScheduler.java:179)
      [org.opennms.core.daemon-28.0.0.jar:?]
             
      at
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      [?:?]
             
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      [?:?]
             
      at
      org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
      [opennms-util-28.0.0.jar:?]
             
      at java.lang.Thread.run(Thread.java:834) [?:?]

      On Sentinel, there is a `sync-node-cache` command.  I am not sure if this is the same cache, or if it's viable to expose this on core, but some mechanism to manually force this cached information to refresh would be beneficial in this case, as restarting the environment to force a refresh takes apprx 40 minutes.

       

      Attachments

        Issue Links

          Activity

            People

              swachter Stefan Wachter
              dino2gnt Dino Yancey
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.