Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-14095

Unable to configure dispatcher threads amount in QueueConfig

    XMLWordPrintable

Details

    • Configuration
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • 27.1.1
    • None
    • Core
    • Security Level: Default (Default Security Scheme)

    Description

      Hi all! We faced this problem while trying to set-up a containerized OpenNMS in an environment which will be primarily for syslog-based fault management. In an already existing deployment we are currently collecting around 20k syslogs per minute, and since that OpenNMS instance is consuming ~6 cores and 12 GB of RAM we configured our new OpenNMS with 8 cores and 16GB of RAM (as we are expecting a pretty similar level of messages).

      Nevertheless, what we saw was that the OpenNMS was unable to process such amount of syslogs. Instead, the messages were getting accumulated in the ActiveMQ, causing an increasing delay in their handling.

      After some tuning and investigation, we think the problem is related with the number of dispatcher threads (OpenNMS.Sink.AsyncDispatcher.Syslog threads in this case) which are continually dequeuing from the message queue. What we saw was that the amount of these threads defaults to 2x<#Cores> (16 threads in our case as we assigned a limit of 8 cores) and we believe this amount cannot be overriden with any configuration.

      We have already compared this amount with our other working deployment and saw that it was running with a CPU limit of 32 cores (resulting in 64 dispatcher threads). When we changed the CPU limit in our new environment to 32 cores as well, the OpenNMS started responding properly and we are now processing ~15k syslogs per minute without problems.

      We believe, however, that this configuration should be available from the XML OpenNMS configuration files, and not as a workaround by changing the CPU limits. We have been observing OpenNMS for a week and it has never consumed more than 8 cores in a week of monitoring. We think being able to configure the amount of syslog dispatcher threads without increasing the application limits is important, as OpenNMS will not likely be the only service running in its host.

      Attachments

        Activity

          People

            dino2gnt Dino Yancey
            agarcia Alejandro Garcia
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.