Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
25.0.0
-
Security Level: Default (Default Security Scheme)
-
Horizon 2019 - October 9th, Horizon 2019 - October 16th, Horizon 2019 - October 23rd
Description
The following exception stack trace appears repeatedly in ipc.log:
java.lang.RuntimeException: javax.xml.bind.UnmarshalException - with linked exception: [Exception [EclipseLink-25004] (Eclipse Persistence Services - 2.5.1.v20130918-f2b9fc5): org.eclipse.persistence.exceptions.XMLMarshalException Exception Description: An error occurred unmarshalling the document Internal Exception: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>. at [row,col {unknown-source}]: [30523,28]] at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:87) ~[org.opennms.core.xml-25.0.0.jar:?] at org.opennms.core.rpc.xml.AbstractXmlRpcModule.unmarshalResponse(AbstractXmlRpcModule.java:81) ~[org.opennms.core.ipc.rpc.xml-25.0.0.jar:?] at org.opennms.core.ipc.rpc.kafka.KafkaRpcClientFactory$ResponseHandler.sendResponse(KafkaRpcClientFactory.java:383) ~[org.opennms.core.ipc.rpc.kafka-25.0.0.jar:?] at org.opennms.core.ipc.rpc.kafka.KafkaRpcClientFactory$KafkaConsumerRunner.lambda$run$0(KafkaRpcClientFactory.java:488) ~[org.opennms.core.ipc.rpc.kafka-25.0.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: javax.xml.bind.UnmarshalException at org.eclipse.persistence.jaxb.JAXBUnmarshaller.handleXMLMarshalException(JAXBUnmarshaller.java:980) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:588) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?] ... 6 more Caused by: org.eclipse.persistence.exceptions.XMLMarshalException: Exception Description: An error occurred unmarshalling the document Internal Exception: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>. at [row,col {unknown-source}]: [30523,28] at org.eclipse.persistence.exceptions.XMLMarshalException.unmarshalException(XMLMarshalException.java:114) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.convertSAXException(SAXUnmarshaller.java:996) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.unmarshal(SAXUnmarshaller.java:903) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.oxm.XMLUnmarshaller.unmarshal(XMLUnmarshaller.java:659) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:585) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?] ... 6 more Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </inst>; expected </instance>. at [row,col {unknown-source}]: [30523,28] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911) ~[woodstox-core-5.0.3.jar:5.0.3] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123) ~[woodstox-core-5.0.3.jar:5.0.3] at org.eclipse.persistence.internal.oxm.record.XMLStreamReaderReader.parse(XMLStreamReaderReader.java:98) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.internal.oxm.record.XMLStreamReaderReader.parse(XMLStreamReaderReader.java:86) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.internal.oxm.record.SAXUnmarshaller.unmarshal(SAXUnmarshaller.java:895) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.oxm.XMLUnmarshaller.unmarshal(XMLUnmarshaller.java:659) ~[org.eclipse.persistence.core-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:585) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.eclipse.persistence.jaxb.JAXBUnmarshaller.unmarshal(JAXBUnmarshaller.java:182) ~[org.eclipse.persistence.moxy-2.5.1.jar:?] at org.opennms.core.xml.XmlHandler.unmarshal(XmlHandler.java:85) ~[org.opennms.core.xml-25.0.0.jar:?] ... 6 more
Indeed, on extracting the 2.1MB response document (available as bad-response-document.xml.gz on support ticket linked in Environment), the problem is plainly visible at line 30523:
<result> <base>.1.3.6.1.2.1.31.1.1.1.8</base> <instance>844</instance> <value type="70">AA==</value> </result> <result> <base>.1.3.6.1.2.1.31.1.1.1.8</base> <instance>845</inst<snmp-response> <response correlation-id="0"> <result> <base>.1.3.6.1.2.1.2.1</base> <instance>0</instance> <value type="2">AVI=</value> </result> </response>
The method in play is KafkaRpcClientFactory.sendResponse.
It looks as if the response document were truncated in mid-stream and its remainder overwritten with another document of the same schema.
I'm awaiting verification from the reporting customer that the Minions are all versioned in lock-step with OpenNMS.