Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8379

Sluggish performance with Cassandra driver

    XMLWordPrintable

Details

    Description

      We've had a few reports of poor performance under high load when using Newts.

      Analyzing the stack traces showed that the several threads were spending alot of their time generating TRACE log messages, which weren't actually being persisted to disk:

      "NewtsWriter-Consumer-7": awaiting notification on [0x00000005c0156980]
      	at sun.misc.Unsafe.park(Native Method)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
      	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367)
      	at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:738)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:63)
      	at com.datastax.driver.core.CodecRegistry.findCodec(CodecRegistry.java:516)
      	at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:392)
      	at com.datastax.driver.core.querybuilder.Utils.appendValue(Utils.java:98)
      	at com.datastax.driver.core.querybuilder.Utils.joinAndAppendValues(Utils.java:62)
      	at com.datastax.driver.core.querybuilder.Insert.buildQueryString(Insert.java:59)
      	at com.datastax.driver.core.querybuilder.BuiltStatement$ForwardingStatement.buildQueryString(BuiltStatement.java:354)
      	at com.datastax.driver.core.querybuilder.Batch.buildQueryString(Batch.java:69)
      	at com.datastax.driver.core.querybuilder.BuiltStatement.toString(BuiltStatement.java:293)
      	at org.apache.logging.log4j.message.ParameterizedMessage.recursiveDeepToString(ParameterizedMessage.java:499)
      	at org.apache.logging.log4j.message.ParameterizedMessage.deepToString(ParameterizedMessage.java:368)
      	at org.apache.logging.log4j.message.ParameterizedMessage.parseArguments(ParameterizedMessage.java:159)
      	at org.apache.logging.log4j.message.ParameterizedMessage.<init>(ParameterizedMessage.java:116)
      	at org.apache.logging.log4j.message.ParameterizedMessageFactory.newMessage(ParameterizedMessageFactory.java:47)
      	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:737)
      	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
      	at org.apache.logging.slf4j.Log4jLogger.trace(Log4jLogger.java:68)
      	at com.datastax.driver.core.RequestHandler.<init>(RequestHandler.java:71)
      	at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
      	at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
      	at org.opennms.newts.cassandra.CassandraSession.execute(CassandraSession.java:94)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:275)
      	at org.opennms.newts.persistence.cassandra.CassandraSampleRepository.insert(CassandraSampleRepository.java:231)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:184)
      	at org.opennms.netmgt.newts.NewtsWriter.onEvent(NewtsWriter.java:70)
      	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:138)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        Activity

          People

            j-white Jesse White
            j-white Jesse White
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: