Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8379

Sluggish performance with Cassandra driver

    XMLWordPrintable

    Details

      Description

      We've had a few reports of poor performance under high load when using Newts.

      Analyzing the stack traces showed that the several threads were spending alot of their time generating TRACE log messages, which weren't actually being persisted to disk:

      "NewtsWriter-Consumer-7": awaiting notification on [0x00000005c0156980]
      	at sun.misc.Unsafe.park(Native Method)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367)
      	at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:738)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:63)
      	at com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:516)
      	at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:392)
      	at com.datastax.driver.core.querybuilder.Utils.appendValue(Utils.java:98)
      	at com.datastax.driver.core.querybuilder.Utils.joinAndAppendValues(Utils.java:62)
      	at com.datastax.driver.core.querybuilder.Insert.buildQueryString(Insert.java:59)
      	at com.datastax.driver.core.querybuilder.BuiltStatement$ForwardingStatement.buildQueryString(BuiltStatement.java:354)
      	at com.datastax.driver.core.querybuilder.Batch.buildQueryString(Batch.java:69)
      	at com.datastax.driver.core.querybuilder.BuiltStatement.toString(BuiltStatement.java:293)
      	at org.apache.logging.log4j.message.ParameterizedMessage.recursiveDeepToString(ParameterizedMessage.java:499)
      	at org.apache.logging.log4j.message.ParameterizedMessage.deepToString(ParameterizedMessage.java:368)
      	at org.apache.logging.log4j.message.ParameterizedMessage.parseArguments(ParameterizedMessage.java:159)
      	at org.apache.logging.log4j.message.ParameterizedMessage.<init>(ParameterizedMessage.java:116)
      	at org.apache.logging.log4j.message.ParameterizedMessageFactory.newMessage(ParameterizedMessageFactory.java:47)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:737)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:68)
      	at com.datastax.driver.core.RequestHandler.<init>(RequestHandler.java:71)
      	at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
      	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
      	at org.opennms.newts.cassandra.CassandraSession.execute(CassandraSession.java:94)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:275)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:231)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:184)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:70)
      	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:138)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            Assignee:
            j-white Jesse White
            Reporter:
            j-white Jesse White
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: