Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13030

When using a custom prefix, the Elasticsearch Forwarder for events and situation-feedback creates a wrong template.

    XMLWordPrintable

Details

    • Backlog CM

    Description

      A customer heavily relies on index prefixes for all the integrations with Elasticsearch because their cluster is shared across multiple different OpenNMS environments.

      When this is the case, the template matching is incorrect, leading to something like this:

      # curl  http://localhost:9200/_cat/templates' 2>/dev/null | grep prod
      prod-eventsindextemplate       [prod-*]                         0
      prod-feedback                  [prod-*]                         0
      prod-alarms                    [prod-opennms-alarms-*]          1
      

      All the Elasticsearch features in OpenNMS were configured with this:

      indexPrefix=prod-
      

      This confuses the system, and the actual indexes could end up with the wrong template.

      The following is the only evidence found in the customer environment proving that the events forwarder is not working:

      2020-11-23T11:49:06,827 | ERROR | EventToIndex-Thread-11 | EventToIndex                     | 421 - org.opennms.features.opennms-es-rest - 26.2.1 | Bulk API action failed. Error response was: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
      

      The karaf.log* files are full of messages like this, as the environment in question processes on average over 300 events per second.

      From the initial list, only the alarms are properly defined. Although, depending on race conditions, the alarms template could end up with the events template and vice-versa, meaning all of them must be fixed.

      Here is what I would expect to see on a healthy system using a prefix:

      prod-eventsindextemplate       [prod-opennms-events-raw-*]                         0
      prod-feedback                  [prod-situation-feedback-*]                         0
      prod-alarms                    [prod-opennms-alarms-*]          1
      

      Attachments

        Issue Links

          Activity

            People

              cgorantla Chandra Gorantla
              agalue Alejandro Galue
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: