Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13211

Improve Event forwarding performance for Kafka producer

    XMLWordPrintable

    Details

    • Type: Enhancement
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Meridian-2020.1.7, 27.1.1
    • Component/s: None
    • Security Level: Default (Default Security Scheme)
    • Labels:
      None
    • Sprint:
      Horizon 2021 - Mar 17 - Mar 31

      Description

      In a deployment handling a large rate of events/s the KafkaProducer threads were found to be stuck with the following stack:

      "org.opennms.features.kafka.producer.OpennmsKafkaProducer-Thread-1-of-4" #229579 prio=5 os_prio=0 cpu=241515665.44ms elapsed=339878.31s tid=0x00007f32e0043000 nid=0x1ae82 runnable  [0x00007f2d51113000]
         java.lang.Thread.State: RUNNABLE
              at org.opennms.netmgt.xml.eventconf.Events.findFirstMatchingEvent(Events.java:319)
              at org.opennms.netmgt.config.DefaultEventConfDao.findByUei(DefaultEventConfDao.java:229)
              at org.opennms.netmgt.config.DefaultEventConfDao.getEventLabel(DefaultEventConfDao.java:147)
              at Proxya49d7288_9cba_44de_90d6_fbd484b3e883.getEventLabel(Unknown Source)
              at org.opennms.features.kafka.producer.ProtobufMapper.toEvent(ProtobufMapper.java:212)
              at org.opennms.features.kafka.producer.OpennmsKafkaProducer.lambda$forwardEvent$4(OpennmsKafkaProducer.java:254)
              at org.opennms.features.kafka.producer.OpennmsKafkaProducer$$Lambda$4506/0x0000000801100440.call(Unknown Source)
              at org.opennms.features.kafka.producer.OpennmsKafkaProducer.sendRecord(OpennmsKafkaProducer.java:385)
              at org.opennms.features.kafka.producer.OpennmsKafkaProducer.forwardEvent(OpennmsKafkaProducer.java:253)
              at org.opennms.features.kafka.producer.OpennmsKafkaProducer.onEvent(OpennmsKafkaProducer.java:478)
              at org.opennms.netmgt.eventd.EventIpcManagerDefaultImpl$EventListenerExecutor$2.run(EventIpcManagerDefaultImpl.java:189)
              at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@11.0.7/CompletableFuture.java:1736)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1128)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
              at org.opennms.core.concurrent.LogPreservingThreadFactory$3.run(LogPreservingThreadFactory.java:124)
              at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
      

        Attachments

          Activity

            People

            Assignee:
            j-white Jesse White
            Reporter:
            j-white Jesse White
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration