Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 17.0.0, 17.1.0, 17.1.1
-
Fix Version/s: 18.0.0, Meridian-2016.1.0
-
Component/s: Data Output - Newts
-
Security Level: Default (Default Security Scheme)
-
Labels:None
Description
We've had a few reports of poor performance under high load when using Newts.
Analyzing the stack traces showed that the several threads were spending alot of their time generating TRACE log messages, which weren't actually being persisted to disk:
"NewtsWriter-Consumer-7": awaiting notification on [0x00000005c0156980] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367) at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112) at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:738) at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708) at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:63) at com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:516) at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:392) at com.datastax.driver.core.querybuilder.Utils.appendValue(Utils.java:98) at com.datastax.driver.core.querybuilder.Utils.joinAndAppendValues(Utils.java:62) at com.datastax.driver.core.querybuilder.Insert.buildQueryString(Insert.java:59) at com.datastax.driver.core.querybuilder.BuiltStatement$ForwardingStatement.buildQueryString(BuiltStatement.java:354) at com.datastax.driver.core.querybuilder.Batch.buildQueryString(Batch.java:69) at com.datastax.driver.core.querybuilder.BuiltStatement.toString(BuiltStatement.java:293) at org.apache.logging.log4j.message.ParameterizedMessage.recursiveDeepToString(ParameterizedMessage.java:499) at org.apache.logging.log4j.message.ParameterizedMessage.deepToString(ParameterizedMessage.java:368) at org.apache.logging.log4j.message.ParameterizedMessage.parseArguments(ParameterizedMessage.java:159) at org.apache.logging.log4j.message.ParameterizedMessage.<init>(ParameterizedMessage.java:116) at org.apache.logging.log4j.message.ParameterizedMessageFactory.newMessage(ParameterizedMessageFactory.java:47) at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:737) at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708) at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:68) at com.datastax.driver.core.RequestHandler.<init>(RequestHandler.java:71) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55) at org.opennms.newts.cassandra.CassandraSession.execute(CassandraSession.java:94) at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:275) at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:231) at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:184) at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:70) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:138) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)