Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-8777

Incoming syslog/trap flood can overwhelm new handler code



    • Horizon - Nov 30th, Horizon - Dec 7th, Horizon - Dec 14th, Horizon - Jan 11th, Horizon - Jan 18th


      If a large backlog of syslog or trap messages generated by a Minion is waiting on a Kafka broker and OpenNMS is started up, it will attempt to stream all of the messages at once into the Camel messaging system. This will exhaust all of the Java heap space and lead to an OutOfMemoryError if the number of messages in the backlog is too large.

      To provide back-pressure on this queue and prevent memory from being exhausted, we should make the incoming Camel SEDA queue have a limited size and mark it as "blockWhenFull".

      It will normally be very close to empty because events will be processed quickly after they are received so putting a limit on the queue size should only come into play when there is a significant backlog of messages that has accumulated in the messaging channel.

      Since most syslog and trap messages will be 1KB - 2KB in size, I would recommend a default queue size of 50,000 which should consume roughly 100MB of RAM under full load.

      This needs to be done in 4 contexts:

      • blueprint-syslog-handler-default.xml
      • blueprint-syslog-handler-kafka-default.xml
      • blueprint-trapd-handler-default.xml
      • blueprint-trapd-handler-kafka-default.xml


        Issue Links



              seth Seth Leger
              seth Seth Leger
              1 Vote for this issue
              3 Start watching this issue