Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8507

upgrade error: too many open files

    Details

      Description

      I was doing an upgrade and got a "too many open files" error:

      Processing DS /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370/ds.properties
      Warning: there are no .meta files on /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370
      Warning: there are no .rrd files on /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370
      Processing DS /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP290/ds.properties
      Warning: can't perform the upgrade operation because: Can't upgrade the JRBs because /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP290/ds.properties (Too many open files)

      • Executing rollback phase
        Warning: can't rollback the upgrade because: Can't restore the backup files because Failed to list contents of /opt/opennms/share/rrd/snmp/7208/jmxJVMIndex/Tomcat8-GRP471
        org.opennms.upgrade.api.OnmsUpgradeException: Can't restore the backup files because Failed to list contents of /opt/opennms/share/rrd/snmp/7208/jmxJVMIndex/Tomcat8-GRP471
        at org.opennms.upgrade.implementations.JmxRrdMigratorOffline.rollback(JmxRrdMigratorOffline.java:186)
        at org.opennms.upgrade.support.Upgrade.executeUpgrade(Upgrade.java:155)
        at org.opennms.upgrade.support.Upgrade.execute(Upgrade.java:247)
        at org.opennms.upgrade.support.Upgrade.main(Upgrade.java:266)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:460)
        at java.lang.Thread.run(Thread.java:745)
      • Running post-execution phase

      After this occurred, the upgrade process still attempted to run addtitional bits that continued to fail with "too many open files"

      Removing backup /opt/opennms/share/rrd/snmp/5041/jmxJVMIndex/GlassFish-GRP200.zip
      Removing backup /opt/opennms/share/rrd/snmp/4521/jmxJVMIndex/GlassFish-GRP259.zip

      Finished in 43 minutes

      Processing JettyConfigMigratorOffline: Adds jetty.xml if necessary: NMS-6629

      • Running pre-execution phase
      • Running execution phase
        SSL Enabled ? true
        AJP Enabled ? false
        Copying /opt/opennms/etc/examples/jetty.xml into /opt/opennms/etc/jetty.xml
        Warning: can't perform the upgrade operation because: Can't fix Jetty configuration because /opt/opennms/etc/examples/jetty.xml (Too many open files)
      • Executing rollback phase
      • Running post-execution phase

      Finished in 0 seconds

      Processing ServiceConfigMigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12: NMS-6970

      • Running pre-execution phase
        Ignoring: Can't load /opt/opennms/jetty-webapps/opennms/WEB-INF/version.properties
        Processing KscReportsMigrator: Fixes the KSC Reports because of SNMP Interface directories changes
      • Running pre-execution phase
        Backing up /opt/opennms/etc/ksc-performance-reports.xml
        Ignoring: Can't initialize ksc-performance-reports.xml because Cannot ZIP files because /opt/opennms/etc/ksc-performance-reports.xml.zip (Too many open files)
        Processing ServiceConfig1701MigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12: NMS-6970
      • Running pre-execution phase
        Backing up /opt/opennms/etc/service-configuration.xml
        Ignoring: Can't backup service-configurations.xml because Cannot ZIP files because /opt/opennms/etc/service-configuration.xml.zip (Too many open files)
        Processing DataCollectionConfigMigratorOffline: Modifies the packages names used in etc/datacollection/*.xml. See NMS-7612.
      • Running pre-execution phase
        Backing up /opt/opennms/etc/datacollection
        java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:460)
        at java.lang.Thread.run(Thread.java:745)
        Caused by: java.lang.NullPointerException
        at org.opennms.upgrade.api.AbstractOnmsUpgrade.populateFilesList(AbstractOnmsUpgrade.java:342)
        at org.opennms.upgrade.api.AbstractOnmsUpgrade.zipDir(AbstractOnmsUpgrade.java:393)
        at org.opennms.upgrade.implementations.DataCollectionConfigMigratorOffline.preExecute(DataCollectionConfigMigratorOffline.java:72)
        at org.opennms.upgrade.support.Upgrade.executeUpgrade(Upgrade.java:141)
        at org.opennms.upgrade.support.Upgrade.execute(Upgrade.java:247)
        at org.opennms.upgrade.support.Upgrade.main(Upgrade.java:266)
        ... 6 more

      There are 725,560 files under /opt/opennms/share/rrd/snmp/.

        Attachments

          Activity

            People

            • Assignee:
              j-white Jesse White
              Reporter:
              ronald.roskens@biworldwide.com Ron Roskens
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: