provisiond hangs

Description

Hi guys,

Provisiond tends to hang quite often here...

Last week a colleague of mine reported me that a new service wasn't discovered by OpenNMS. It quickly turned out that provisiond was hung. After an OpenNMS restart and an upgrade to 1.10.6-0.20120918.1 OpenNMS started to try discovering services again.

Yesterday, the same colleague called again; provisiond seems hung again.

I can't find any relevant message in provisiond.log, neither in output.log, misc.log or controller.log.

"service opennms status" doesn't report any problem:

support@opennms:~$ sudo service opennms status
OpenNMS.Eventd : running
OpenNMS.Trapd : running
OpenNMS.Queued : running
OpenNMS.Dhcpd : running
OpenNMS.Actiond : running
OpenNMS.Notifd : running
OpenNMS.Scriptd : running
OpenNMS.Rtcd : running
OpenNMS.Pollerd : running
OpenNMS.PollerBackEnd : running
OpenNMS.Collectd : running
OpenNMS.Threshd : running
OpenNMS.Vacuumd : running
OpenNMS.EventTranslator: running
OpenNMS.PassiveStatusd : running
OpenNMS.Statsd : running
OpenNMS.Provisiond : running
OpenNMS.Reportd : running
OpenNMS.Alarmd : running
OpenNMS.Ackd : running
OpenNMS.JettyServer : running
OpenNMS.Linkd : running
OpenNMS.Syslogd : running
opennms is running
support@opennms:~$

I've attached a screenshot of the node's page showing that I've tried 2 forced rescan yesterday with no result until now

Can you help me?

Cyrille

Environment

Debian 6.0 amd64 LANG=fr_BE.UTF-8 JVM: openjdk-6-jre 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime, using Hotspot JIT

Acceptance / Success Criteria

None

Attachments

6

Lucidchart Diagrams

Activity

Show:

Seth Leger September 6, 2017 at 2:36 PM

I didn't realize that the stack dumps were that old when I reviewed this issue last year.

Let's go ahead and mark this as fixed. There have been many concurrency and resource bugfixes in Provisiond since this was opened.

Cyrille Bollu September 4, 2017 at 6:46 AM

Hello Ben, my original report is quite old and I haven't see this bug for a while (>1year at least). So, for what I'm concerned, it's solved

I'm not running 20.0.2 though (but rather 19.1.0).

BR

Benjamin Reed August 30, 2017 at 7:40 PM

A couple of provisiond-related resource issues were fixed in 20.0.2 – can you confirm if this still happens?

Doug Hoffman July 14, 2016 at 4:07 PM

To be clear, I see provisiond repeatedly firing off SYNs for the same ports on one node when the SYNs are dropped. Observed in our production environment with the following detectors: ftp, smtp, pop3, imap, tcp.

Doug Hoffman July 14, 2016 at 3:55 PM

Yes, but set to ipMatch '127.0.0.1'.

Fixed

Details

Assignee

Reporter

Components

Affects versions

Priority

PagerDuty

Created October 23, 2012 at 4:42 AM
Updated September 6, 2017 at 2:36 PM
Resolved September 6, 2017 at 2:36 PM