Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-12784

Kafka Producer puts all events on the same partition when using `donotpersist`

    XMLWordPrintable

    Details

    • Sprint:
      Horizon 2020 - June 24

      Description

      I've been helping a customer building one of their production environments, and they heavily rely on the Kafka Producer.

      Without entering into details, OpenNMS is receiving Syslog and SNMP Traps from multiple devices via Minions installed on different data centers. The goal for the majority of the events is forward them to Kafka via the producer, but do not persist them to the database, to avoid making PostgreSQL a bottleneck on their environment.

      Unfortunately, the logic uses the Event ID as the Partition Key when forwarding events, and that only works when the event is persisted in the database. When the event definition has donotpersist, OpenNMS can still process it, and the Kafka Producer will forward it as expected. The problem is that the Message Key will always be 0 in this case, meaning all the messages will end into the same partition, making it impossible to scale out the application.

      For this reason, we need to find something unique and use that as the Key. For alarms, we use the reduction-key, but unfortunately, there is no obvious field in terms of events.

        Attachments

          Activity

            People

            Assignee:
            cgorantla Chandra Gorantla
            Reporter:
            agalue Alejandro Galue
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: