Fixed
Details
Assignee
Jesse WhiteJesse WhiteReporter
Jeff GehlbachJeff GehlbachLabels
Components
Sprint
NoneFix versions
Affects versions
Priority
Critical
Details
Details
Assignee
Jesse White
Jesse WhiteReporter
Jeff Gehlbach
Jeff GehlbachLabels
Components
Sprint
None
Fix versions
Affects versions
Priority
PagerDuty
PagerDuty
PagerDuty
Created October 31, 2016 at 11:24 AM
Updated March 1, 2017 at 9:47 AM
Resolved March 1, 2017 at 9:47 AM
To reproduce this issue without a ton of actual RRD files, just make a bazillion ~40KB garbage files beneath
share/rrd/snmp
. Half a million files ought to be a good start. I'm not exaggerating – this kind of scale exists in the wild.Run the installer. You'll see it get through most of its work, then apparently hang after printing:
(The last messages printed may vary from one release to the next)
You'll see the logs go silent except for gc.log (if you've got that one configured) and maybe output.log. If you run
lsof
during this period, you'll see that the JVM is opening each of those files. I've seen this go on for hours, which is a big deal on production systems.This behavior is apparently down to the way Spring does class loading, and it's scary difficult to fix. Fortunately there is a workaround: temporarily remove the symbolic link named
share
fromOPENNMS_HOME
, and restore it after the installer completes but before starting OpenNMS again. Assuming you installed from packages, this trick is available. Just be sure to note the path that the symlink pointed to before removing it (it may not point where you think it does, especially on RPM-based installs where the path is relocatable).