Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8777

Incoming syslog/trap flood can overwhelm new handler code

    XMLWordPrintable

    Details

    • Sprint:
      Horizon - Nov 30th, Horizon - Dec 7th, Horizon - Dec 14th, Horizon - Jan 11th, Horizon - Jan 18th

      Description

      If a large backlog of syslog or trap messages generated by a Minion is waiting on a Kafka broker and OpenNMS is started up, it will attempt to stream all of the messages at once into the Camel messaging system. This will exhaust all of the Java heap space and lead to an OutOfMemoryError if the number of messages in the backlog is too large.

      To provide back-pressure on this queue and prevent memory from being exhausted, we should make the incoming Camel SEDA queue have a limited size and mark it as "blockWhenFull".

      It will normally be very close to empty because events will be processed quickly after they are received so putting a limit on the queue size should only come into play when there is a significant backlog of messages that has accumulated in the messaging channel.

      Since most syslog and trap messages will be 1KB - 2KB in size, I would recommend a default queue size of 50,000 which should consume roughly 100MB of RAM under full load.

      This needs to be done in 4 contexts:

      • blueprint-syslog-handler-default.xml
      • blueprint-syslog-handler-kafka-default.xml
      • blueprint-trapd-handler-default.xml
      • blueprint-trapd-handler-kafka-default.xml

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                seth Seth Leger
                Reporter:
                seth Seth Leger
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: