upgrade error: too many open files

Description

I was doing an upgrade and got a "too many open files" error:

Processing DS /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370/ds.properties
Warning: there are no .meta files on /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370
Warning: there are no .rrd files on /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP370
Processing DS /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP290/ds.properties
Warning: can't perform the upgrade operation because: Can't upgrade the JRBs because /opt/opennms/share/rrd/snmp/5939/jmxJVMIndex/GlassFish-GRP290/ds.properties (Too many open files)

  • Executing rollback phase
    Warning: can't rollback the upgrade because: Can't restore the backup files because Failed to list contents of /opt/opennms/share/rrd/snmp/7208/jmxJVMIndex/Tomcat8-GRP471
    org.opennms.upgrade.api.OnmsUpgradeException: Can't restore the backup files because Failed to list contents of /opt/opennms/share/rrd/snmp/7208/jmxJVMIndex/Tomcat8-GRP471
    at org.opennms.upgrade.implementations.JmxRrdMigratorOffline.rollback(JmxRrdMigratorOffline.java:186)
    at org.opennms.upgrade.support.Upgrade.executeUpgrade(Upgrade.java:155)
    at org.opennms.upgrade.support.Upgrade.execute(Upgrade.java:247)
    at org.opennms.upgrade.support.Upgrade.main(Upgrade.java:266)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:460)
    at java.lang.Thread.run(Thread.java:745)

  • Running post-execution phase

After this occurred, the upgrade process still attempted to run addtitional bits that continued to fail with "too many open files"

Removing backup /opt/opennms/share/rrd/snmp/5041/jmxJVMIndex/GlassFish-GRP200.zip
Removing backup /opt/opennms/share/rrd/snmp/4521/jmxJVMIndex/GlassFish-GRP259.zip

Finished in 43 minutes

Processing JettyConfigMigratorOffline: Adds jetty.xml if necessary:

  • Running pre-execution phase

  • Running execution phase
    SSL Enabled ? true
    AJP Enabled ? false
    Copying /opt/opennms/etc/examples/jetty.xml into /opt/opennms/etc/jetty.xml
    Warning: can't perform the upgrade operation because: Can't fix Jetty configuration because /opt/opennms/etc/examples/jetty.xml (Too many open files)

  • Executing rollback phase

  • Running post-execution phase

Finished in 0 seconds

Processing ServiceConfigMigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12:

  • Running pre-execution phase
    Ignoring: Can't load /opt/opennms/jetty-webapps/opennms/WEB-INF/version.properties
    Processing KscReportsMigrator: Fixes the KSC Reports because of SNMP Interface directories changes

  • Running pre-execution phase
    Backing up /opt/opennms/etc/ksc-performance-reports.xml
    Ignoring: Can't initialize ksc-performance-reports.xml because Cannot ZIP files because /opt/opennms/etc/ksc-performance-reports.xml.zip (Too many open files)
    Processing ServiceConfig1701MigratorOffline: Fixes service-configuration.xml if necessary when upgrading from 1.12:

  • Running pre-execution phase
    Backing up /opt/opennms/etc/service-configuration.xml
    Ignoring: Can't backup service-configurations.xml because Cannot ZIP files because /opt/opennms/etc/service-configuration.xml.zip (Too many open files)
    Processing DataCollectionConfigMigratorOffline: Modifies the packages names used in etc/datacollection/*.xml. See .

  • Running pre-execution phase
    Backing up /opt/opennms/etc/datacollection
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.opennms.bootstrap.Bootstrap$4.run(Bootstrap.java:460)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.NullPointerException
    at org.opennms.upgrade.api.AbstractOnmsUpgrade.populateFilesList(AbstractOnmsUpgrade.java:342)
    at org.opennms.upgrade.api.AbstractOnmsUpgrade.zipDir(AbstractOnmsUpgrade.java:393)
    at org.opennms.upgrade.implementations.DataCollectionConfigMigratorOffline.preExecute(DataCollectionConfigMigratorOffline.java:72)
    at org.opennms.upgrade.support.Upgrade.executeUpgrade(Upgrade.java:141)
    at org.opennms.upgrade.support.Upgrade.execute(Upgrade.java:247)
    at org.opennms.upgrade.support.Upgrade.main(Upgrade.java:266)
    ... 6 more

There are 725,560 files under /opt/opennms/share/rrd/snmp/.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Jesse White July 26, 2016 at 8:06 AM

Cherry picked to foundation-2016.

Jesse White July 26, 2016 at 8:04 AM

Fixed

Details

Assignee

Reporter

Components

Affects versions

Priority

PagerDuty

Created June 17, 2016 at 1:03 PM
Updated July 26, 2016 at 12:05 PM
Resolved July 26, 2016 at 8:05 AM