Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8860

DB installer tries to class-load every file under OPENNMS_HOME as a JAR

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 18.0.2, Meridian-2015.1.3, Meridian-2016.1.3
    • Fix Version/s: 19.1.0, Meridian-2017.1.0
    • Component/s: Installation
    • Security Level: Default (Default Security Scheme)
    • Labels:
    • Environment:
      Any system with RRDTool or JRobin persisting and many (many, many) RRD files in {{OPENNMS_HOME/share/rrd}}, especially on rotational disks
    • Sprint:
      Horizon - Feb 22nd

      Description

      To reproduce this issue without a ton of actual RRD files, just make a bazillion ~40KB garbage files beneath share/rrd/snmp. Half a million files ought to be a good start. I'm not exaggerating – this kind of scale exists in the wild.

      Run the installer. You'll see it get through most of its work, then apparently hang after printing:

      - checking if database "opennms" is unicode... ALREADY UNICODE
      - Checking for old import files in /opt/opennms/etc... DONE
      

      (The last messages printed may vary from one release to the next)

      You'll see the logs go silent except for gc.log (if you've got that one configured) and maybe output.log. If you run lsof during this period, you'll see that the JVM is opening each of those files. I've seen this go on for hours, which is a big deal on production systems.

      This behavior is apparently down to the way Spring does class loading, and it's scary difficult to fix. Fortunately there is a workaround: temporarily remove the symbolic link named share from OPENNMS_HOME, and restore it after the installer completes but before starting OpenNMS again. Assuming you installed from packages, this trick is available. Just be sure to note the path that the symlink pointed to before removing it (it may not point where you think it does, especially on RPM-based installs where the path is relocatable).

        Attachments

          Activity

            People

            • Assignee:
              j-white Jesse White
              Reporter:
              jeffg Jeff Gehlbach
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: