Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-10720

Make Events immutable (avoid CMEs and fix non-deterministic behavior)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Meridian-2018.1.8
    • Fix Version/s: 26.0.0
    • Component/s: Correlator, Events
    • Security Level: Default (Default Security Scheme)
    • Labels:
      None
    • Environment:
      RHEL 6.10 x86_64
      Oracle JDK 1.8.0_191
    • Sprint:
      Horizon 2020 - January 8th

      Description

      A Drools rule caught a ConcurrentModificationException while marshaling an event using JaxbUtils.marshal(Event event).

      No other Drools engines are subscribed to this UEI, and its logmsg.dest is donotpersist, so nothing else should be modifying the event.

      Note that the Event.toString() method in the catch statement did not throw an exception.

      The rule in question:

      rule "CloneNewEvents"
    salience 999999
    lock-on-active
    when
        $event : Event($uei : uei)
    then
        String logTag = "nodeProvisioning:cloneEvent ::";
        LOG.debug("{} Deleting and cloning event with uei {}", logTag, $uei);
        delete($event);
        try {
            String strEvent = JaxbUtils.marshal($event);
            Event clonedEvent = JaxbUtils.unmarshal(Event.class, Sanitation.stripNonValidXMLCharactersByRegex(strEvent));
            insert(clonedEvent);
        } catch (Exception ex) {
            LOG.error("{} Exception caught while cloning event {}:\n", logTag, $event.toString(), ex);
        }
    end

      The exception, with the event's details redacted:

      2019-05-29 05:16:47,723 ERROR [DroolsCorrelationEngine-Notify-Thread] D.Engine: nodeProvisioning:cloneEvent :: Exception caught while cloning event [[ JSONified event ]]:

org.opennms.core.xml.MarshallingResourceFailureException: Failed to marshal/unmarshal XML file while marshalling Event: javax.xml.bind.MarshalException
 - with linked exception:
[java.util.ConcurrentModificationException]; nested exception is javax.xml.bind.MarshalException
 - with linked exception:
[java.util.ConcurrentModificationException]
    at org.opennms.core.xml.MarshallingExceptionTranslator.translate(MarshallingExceptionTranslator.java:61) ~[org.opennms.core.xml-2018.1.8.jar:?]
    at org.opennms.core.xml.JaxbUtils.marshal(JaxbUtils.java:182) ~[org.opennms.core.xml-2018.1.8.jar:?]
    at org.opennms.core.xml.JaxbUtils.marshal(JaxbUtils.java:117) ~[org.opennms.core.xml-2018.1.8.jar:?]
    at org.opennms.netmgt.correlation.drools.Rule_CloneNewEvents777744953.defaultConsequence(Rule_CloneNewEvents777744953.java:11) [drools-core-7.7.0.Final.jar:?]
    at org.opennms.netmgt.correlation.drools.Rule_CloneNewEvents777744953DefaultConsequenceInvokerGenerated.evaluate(Unknown Source) [?:?]
    at org.opennms.netmgt.correlation.drools.Rule_CloneNewEvents777744953DefaultConsequenceInvoker.evaluate(Unknown Source) [drools-core-7.7.0.Final.jar:?]
    at org.drools.core.phreak.RuleExecutor.innerFireActivation(RuleExecutor.java:431) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.phreak.RuleExecutor.fireActivation(RuleExecutor.java:379) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.phreak.RuleExecutor.fire(RuleExecutor.java:135) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:88) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.concurrent.AbstractRuleEvaluator.internalEvaluateAndFire(AbstractRuleEvaluator.java:34) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.concurrent.SequentialRuleEvaluator.evaluateAndFire(SequentialRuleEvaluator.java:43) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1067) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.common.DefaultAgenda.internalFireAllRules(DefaultAgenda.java:1014) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1006) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1308) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1299) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1283) [drools-core-7.7.0.Final.jar:7.7.0.Final]
    at org.opennms.netmgt.correlation.drools.DroolsCorrelationEngine.correlate(DroolsCorrelationEngine.java:136) [drools-correlation-engine-2018.1.8.jar:?]
    at org.opennms.netmgt.correlation.Correlator$EngineAdapter.onEvent(Correlator.java:91) [opennms-correlator-2018.1.8.jar:?]
    at org.opennms.netmgt.eventd.EventIpcManagerDefaultImpl$EventListenerExecutor$2.run(EventIpcManagerDefaultImpl.java:189) [org.opennms.features.events.daemon-2018.1.8.jar:?]
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) [?:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
    at org.opennms.core.concurrent.LogPreservingThreadFactory$2.run(LogPreservingThreadFactory.java:106) [opennms-util-2018.1.8.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: javax.xml.bind.MarshalException
    at org.eclipse.persistence.jaxb.JAXBMarshaller.marshal(JAXBMarshaller.java:686) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
    at org.opennms.core.xml.JaxbUtils.marshal(JaxbUtils.java:180) ~[org.opennms.core.xml-2018.1.8.jar:?]
    ... 24 more
Caused by: java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) ~[?:1.8.0_191]
    at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[?:1.8.0_191]
    at org.eclipse.persistence.internal.queries.InterfaceContainerPolicy.next(InterfaceContainerPolicy.java:274) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.queries.ContainerPolicy.next(ContainerPolicy.java:1166) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.queries.ContainerPolicy.next(ContainerPolicy.java:1) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XMLCompositeCollectionMappingNodeValue.marshal(XMLCompositeCollectionMappingNodeValue.java:107) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.NodeValue.marshal(NodeValue.java:149) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.NodeValue.marshal(NodeValue.java:102) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.record.ObjectMarshalContext.marshal(ObjectMarshalContext.java:59) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XPathNode.marshal(XPathNode.java:401) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XPathNode.marshal(XPathNode.java:382) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XPathObjectBuilder.buildRow(XPathObjectBuilder.java:240) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.TreeObjectBuilder.buildRow(TreeObjectBuilder.java:118) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.TreeObjectBuilder.buildRow(TreeObjectBuilder.java:1) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XMLMarshaller.marshal(XMLMarshaller.java:751) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XMLMarshaller.marshalStreamOrWriter(XMLMarshaller.java:1128) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XMLMarshaller.marshal(XMLMarshaller.java:1079) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.internal.oxm.XMLMarshaller.marshal(XMLMarshaller.java:1034) ~[org.eclipse.persistence.core-2.5.1.jar:?]
    at org.eclipse.persistence.jaxb.JAXBMarshaller.marshal(JAXBMarshaller.java:684) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
    at org.opennms.core.xml.JaxbUtils.marshal(JaxbUtils.java:180) ~[org.opennms.core.xml-2018.1.8.jar:?]
    ... 24 more

        Attachments

          Activity

            People

            Assignee:
            pierreb Pierre Bouffard
            Reporter:
            wkeaney Will Keaney
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: