Runaway threads consuming CPU when rendering certain graphs

Description

When attempting to display a specific graph I've created for Netscreen memory utilization, I see the following symptoms:

  • Page renders everything but the busted graph, and browser shows that the page is still loading, seemingly forever (at least a few minutes)

  • Reloading the page does not help, and in fact 'top' shows that Java is eating a single CPU's worth of resources for each time I reload and trigger another hung/spinning thread

  • Deleting the .jrb file and allowing it to be recreated also does not help

I restarted OpenNMS and then triggered the condition via page reload, taking thread dumps at 1, 2, 3, 4, 5, 10, 15, and 20 hung threads. That data is attached in the "thread dumps" zipfile.

I have also attached the corresponding .jrb files for three Netscreen nodes that all exhibit this behavior. I'm using store by group and have included the ds.properties file, as well as the graph definition (report.juniper.netscreen.host.memory is the glitched graph).

Environment

OpenNMS 1.9.90 snapshot from 2011-08-07 CentOS 5.5 Dedicated Dell PE M610 w/12 cores of Xeon X5650 and 48 GB RAM running OpenNMS + PostgreSQL 9.0 Oracle Java JDK 6u26 19 managed nodes

Acceptance / Success Criteria

None

Attachments

3
  • 08 Sep 2011, 04:22 PM
  • 08 Aug 2011, 06:05 PM
  • 08 Aug 2011, 06:05 PM

Lucidchart Diagrams

Activity

Show:

Andy Ellsworth September 14, 2011 at 10:53 PM

Confirmed fixed in 1.9.91. Thanks!

Donald Desloge September 8, 2011 at 4:22 PM

Patch for JRobin, fixes infinite loop occurrence when range is too large.

Matt Brozowski September 7, 2011 at 11:49 AM

The bug appears to be in JRobin...

Donald is going to debug and try to fix

Matt Brozowski September 7, 2011 at 11:48 AM

I have created a test to reproduce this and checked it into DefaultRrdDaoIntegrationTest but prefixed with xxx since it fails.

Revision: 96b1d0691b8a5d69627f52e287118fab8d6ffe94

Matt Brozowski September 7, 2011 at 11:47 AM

I am working on a test to reproduce this

Fixed

Details

Assignee

Reporter

Labels

Fix versions

Affects versions

Priority

PagerDuty

Created August 8, 2011 at 6:05 PM
Updated January 27, 2017 at 4:20 PM
Resolved September 13, 2011 at 3:56 PM

Flag notifications