Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Possible FileDescriptor Leak in Provisiond

Description

Several days in a row, now, I've seen OpenNMS seem to just... lose all its connections to Postgres. pg_top shows no connections at all. A lot of exceptions in log files about database issues, and the UI just gives lots of 404 errors.

Restarting OpenNMS seems to get everything going again.

Am I maybe missing some bit of Postgres tuning?

I've attached the jstack output from before me restarting OpenNMS this last time; I can try to dig up log entries that may be relevant as well.

Acceptance / Success Criteria

None

Attachments

4
  • 08 Aug 2017, 11:01 PM
  • 01 Aug 2017, 11:23 PM
  • 01 Aug 2017, 08:30 PM
  • 28 Jul 2017, 09:24 PM

Lucidchart Diagrams

Activity

Show:

Jesse White August 15, 2017 at 12:52 PM

Jesse White August 15, 2017 at 12:52 PM

Upgraded Mina to 2.0.16 in foundation-2017 and cherry-picked to release-20.0.2.

Mike Kelly August 10, 2017 at 10:13 AM

It's also possible this could be considered a regression from https://opennms.atlassian.net/browse/NMS-8618#icft=NMS-8618 ?

Mike Kelly August 10, 2017 at 10:12 AM

After discussion in Mattermost, I tried upgrading to mina-core-2.0.16.jar, and that seems to have solved this issue. I see no detectable "jumps" in the open FDs graph anymore.

Diffing `lsof` output between when I restarted OpenNMS after that JAR file replacement and now shows no leakage of "eventpoll" or "pipe" filehandles.

Mike Kelly August 8, 2017 at 11:02 PM

I've attached the log generated by http://file-leak-detector.kohsuke.org/ with threshold=5000.

Fixed

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

PagerDuty

Created July 28, 2017 at 9:24 PM
Updated August 15, 2017 at 12:52 PM
Resolved August 15, 2017 at 12:52 PM

Flag notifications