Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8379

Sluggish performance with Cassandra driver

    XMLWordPrintable

    Details

      Description

      We've had a few reports of poor performance under high load when using Newts.

      Analyzing the stack traces showed that the several threads were spending alot of their time generating TRACE log messages, which weren't actually being persisted to disk:

      "NewtsWriter-Consumer-7": awaiting notification on [0x00000005c0156980]
      	at sun.misc.Unsafe.park(Native Method)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367)
      	at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:738)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:63)
      	at com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:516)
      	at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:392)
      	at com.datastax.driver.core.querybuilder.Utils.appendValue(Utils.java:98)
      	at com.datastax.driver.core.querybuilder.Utils.joinAndAppendValues(Utils.java:62)
      	at com.datastax.driver.core.querybuilder.Insert.buildQueryString(Insert.java:59)
      	at com.datastax.driver.core.querybuilder.BuiltStatement$ForwardingStatement.buildQueryString(BuiltStatement.java:354)
      	at com.datastax.driver.core.querybuilder.Batch.buildQueryString(Batch.java:69)
      	at com.datastax.driver.core.querybuilder.BuiltStatement.toString(BuiltStatement.java:293)
      	at org.apache.logging.log4j.message.ParameterizedMessage.recursiveDeepToString(ParameterizedMessage.java:499)
      	at org.apache.logging.log4j.message.ParameterizedMessage.deepToString(ParameterizedMessage.java:368)
      	at org.apache.logging.log4j.message.ParameterizedMessage.parseArguments(ParameterizedMessage.java:159)
      	at org.apache.logging.log4j.message.ParameterizedMessage.<init>(ParameterizedMessage.java:116)
      	at org.apache.logging.log4j.message.ParameterizedMessageFactory.newMessage(ParameterizedMessageFactory.java:47)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:737)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:68)
      	at com.datastax.driver.core.RequestHandler.<init>(RequestHandler.java:71)
      	at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
      	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
      	at org.opennms.newts.cassandra.CassandraSession.execute(CassandraSession.java:94)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:275)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:231)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:184)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:70)
      	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:138)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              j-white Jesse White
              Reporter:
              j-white Jesse White
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: