Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13017

When using a custom prefix, the Elasticsearch Forwarder for events and situation-feedback creates a wrong template.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 26.2.1, Meridian-2020.1.1, 27.0.0
    • Fix Version/s: 27.0.1
    • Component/s: Alarms, Events
    • Security Level: Default (Default Security Scheme)
    • Labels:
    • Sprint:
      Horizon 2020 - Nov 11-Nov 24, Horizon 2020 - Nov 24-Dec 9
    • HB Backlog Status:
      Backlog CM

      Description

      A customer heavily relies on index prefixes for all the integrations with Elasticsearch because their cluster is shared across multiple different OpenNMS environments.

      When this is the case, the template matching is incorrect, leading to something like this:

      # curl  http://localhost:9200/_cat/templates' 2>/dev/null | grep prod
      prod-eventsindextemplate       [prod-*]                         0
      prod-feedback                  [prod-*]                         0
      prod-alarms                    [prod-opennms-alarms-*]          1
      

      All the Elasticsearch features in OpenNMS were configured with this:

      indexPrefix=prod-
      

      This confuses the system, and the actual indexes could end up with the wrong template.

      The following is the only evidence found in the customer environment proving that the events forwarder is not working:

      2020-11-23T11:49:06,827 | ERROR | EventToIndex-Thread-11 | EventToIndex                     | 421 - org.opennms.features.opennms-es-rest - 26.2.1 | Bulk API action failed. Error response was: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
      

      The karaf.log* files are full of messages like this, as the environment in question processes on average over 300 events per second.

      From the initial list, only the alarms are properly defined. Although, depending on race conditions, the alarms template could end up with the events template and vice-versa, meaning all of them must be fixed.

      Here is what I would expect to see on a healthy system using a prefix:

      prod-eventsindextemplate       [prod-opennms-events-raw-*]                         0
      prod-feedback                  [prod-situation-feedback-*]                         0
      prod-alarms                    [prod-opennms-alarms-*]          1
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              cgorantla Chandra Gorantla
              Reporter:
              agalue Alejandro Galue
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration