Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 25.0.0
-
Fix Version/s: 25.1.0
-
Component/s: Data Collection - SNMP, Minion
-
Security Level: Default (Default Security Scheme)
-
Labels:
-
Environment:
-
Sprint:Horizon 2019 - October 9th, Horizon 2019 - October 16th, Horizon 2019 - October 23rd
Description
The following exception stack trace appears repeatedly in ipc.log:
java.lang.RuntimeException: javax.xml.bind.UnmarshalException
- with linked exception:
[Exception [EclipseLink-25004] (Eclipse Persistence Services - 2.5.1.v20130918-f2b9fc5): org.eclipse.persistence.exceptions.XMLMarshalException
Exception Description: An error occurred unmarshalling the document
Internal Exception: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>.
at [row,col {unknown-source}]: [30523,28]]
at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:87) ~[org.opennms.core.xml-25.0.0.jar:?]
at org.opennms.core.rpc.xml.AbstractXmlRpcModule.unmarshalResponse(AbstractXmlRpcModule.java:81) ~[org.opennms.core.ipc.rpc.xml-25.0.0.jar:?]
at org.opennms.core.ipc.rpc.kafka.KafkaRpcClientFactory$ResponseHandler.sendResponse(KafkaRpcClientFactory.java:383) ~[org.opennms.core.ipc.rpc.kafka-25.0.0.jar:?]
at org.opennms.core.ipc.rpc.kafka.KafkaRpcClientFactory$KafkaConsumerRunner.lambda$run$0(KafkaRpcClientFactory.java:488) ~[org.opennms.core.ipc.rpc.kafka-25.0.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.xml.bind.UnmarshalException
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.handleXMLMarshalException(JAXBUnmarshaller.java:980) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:588) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?]
... 6 more
Caused by: org.eclipse.persistence.exceptions.XMLMarshalException:
Exception Description: An error occurred unmarshalling the document
Internal Exception: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>.
at [row,col {unknown-source}]: [30523,28]
at org.eclipse.persistence.exceptions.XMLMarshalException.unmarshalException(XMLMarshalException.java:114) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.convertSAXException(SAXUnmarshaller.java:996) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.unmarshal(SAXUnmarshaller.java:903) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.oxm.XMLUnmarshaller.unmarshal(XMLUnmarshaller.java:659) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:585) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?]
... 6 more
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>.
at [row,col {unknown-source}]: [30523,28]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911) ~[woodstox-core-5.0.3.jar:5.0.3]
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123) ~[woodstox-core-5.0.3.jar:5.0.3]
at org.eclipse.persistence.internal.oxm.record.XMLStreamReaderReader.parse(XMLStreamReaderReader.java:98) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.internal.oxm.record.XMLStreamReaderReader.parse(XMLStreamReaderReader.java:86) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.unmarshal(SAXUnmarshaller.java:895) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.oxm.XMLUnmarshaller.unmarshal(XMLUnmarshaller.java:659) ~[org.eclipse.persistence.core-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:585) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?]
at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?]
... 6 more
Indeed, on extracting the 2.1MB response document (available as bad-response-document.xml.gz on support ticket linked in Environment), the problem is plainly visible at line 30523:
<result>
<base>.1.3.6.1.2.1.31.1.1.1.8</base>
<instance>844</instance>
<value type="70">AA==</value>
</result>
<result>
<base>.1.3.6.1.2.1.31.1.1.1.8</base>
<instance>845</inst<snmp-response>
<response correlation-id="0">
<result>
<base>.1.3.6.1.2.1.2.1</base>
<instance>0</instance>
<value type="2">AVI=</value>
</result>
</response>
The method in play is KafkaRpcClientFactory.sendResponse.
It looks as if the response document were truncated in mid-stream and its remainder overwritten with another document of the same schema.
I'm awaiting verification from the reporting customer that the Minions are all versioned in lock-step with OpenNMS.
