Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-10310

Kafka consumers on Sentinel are not shown on kafka-manager

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 23.0.0
    • Fix Version/s: 23.0.0
    • Component/s: Architecture
    • Security Level: Default (Default Security Scheme)
    • Labels:
    • Environment:
      Running latest RPMs from the develop branch on the Core OpenNMS, and latest RPMs from the features/sentinel branch on 2 Sentinel VMs

      Description

      I was trying latest development snapshot of Sentinel, from its RPMs, and its Docker image through Kubernetes:

      https://github.com/OpenNMS/opennms-drift-aws/tree/release/horizon-23

      https://github.com/OpenNMS/opennms-drift-kubernetes/tree/release/horizon-23

      In both cases, I saw that the solution appears to be working as expected, as I can see flow data on Elasticsearch, and Telemetryd is not handling flows on the OpenNMS Core instance; only Sentinel instances are processing flow data.

      Typically, when debugging Kafka clients, I use Kafka Manager (a tool from Yahoo https://github.com/yahoo/kafka-manager), to inspect the Kafka cluster, see the amount of messages per topic, see the number of consumers currently connected and details about them, message lag, etc.

      This tools shows the consumers from OpenNMS for Sink, RPC and Flows when Telemetryd is handling flows within OpenNMS. Now, when using Sentinel, the consumers for the flow topics are not shown.

      Interestingly, if you manually check Zookeeper, or use the kafka-consumer-groups.sh command on any Kafka node, you can see the consumers, but after specifying additional flags; as they are not registered on the usual way.

      For example:

      The following command returns nothing:

      /opt/kafka/bin/kafka-consumer-groups.sh --describe --group Sentinel --bootstrap-server kafka1:9092 

      But, the following returns some information:

      [ec2-user@kafka1 bin]$ /opt/kafka/bin/kafka-consumer-groups.sh --describe --group Sentinel --bootstrap-server kafka1:9092 --members --verbose
      
      CONSUMER-ID                                      HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT
      consumer-3-770960d3-fc8f-4843-b7f6-a564fb48871f  /18.191.19.2    consumer-3      4               OpenNMS.Sink.Telemetry-Netflow-9(8,9,10,11)
      consumer-2-79f33919-4971-4574-8e76-5e30cc5e7264  /18.191.19.2    consumer-2      4               OpenNMS.Sink.Telemetry-Netflow-5(8,9,10,11)
      consumer-13-9f5ea5a7-44c0-4e3c-af4f-bcc33954b307 /18.191.19.2    consumer-13     4               OpenNMS.Sink.Telemetry-Netflow-9(4,5,6,7)
      consumer-7-e3f51ab9-622b-4a5e-8b2a-a9c360db09ab  /18.191.19.2    consumer-7      4               OpenNMS.Sink.Telemetry-Netflow-9(12,13,14,15)
      consumer-5-478e9b87-0665-4c40-b69b-2426c2ae81f1  /18.191.19.2    consumer-5      4               OpenNMS.Sink.Telemetry-Netflow-5(12,13,14,15)
      consumer-10-22ac5f9b-5a9b-4aa9-8d4c-90cd75ec7345 /18.191.19.2    consumer-10     4               OpenNMS.Sink.Telemetry-Netflow-9(0,1,2,3)
      consumer-11-d18cd114-a3a1-4dd2-a950-a53adf17ff7f /18.191.19.2    consumer-11     4               OpenNMS.Sink.Telemetry-Netflow-5(0,1,2,3)
      consumer-1-f9928728-48e9-4ef1-8713-e8e4574b71ba  /18.191.19.2    consumer-1      50              __consumer_offsets(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49)
      consumer-16-f0da778a-ff46-4558-a768-04198b833b35 /18.191.19.2    consumer-16     4               OpenNMS.Sink.Telemetry-Netflow-5(4,5,6,7) 

      Note that --members has been passed in order to see the data.

      If you just list the consumer groups, the Sentinel group is shown (also in kafka-manager), but no client information is displayed.

      I've been using Kafka Manager a lot, not only with OpenNMS, but also for personal projects, and I've never seen this behavior before with consumer groups.

      I can provide more details if required, and I can even provide access to any of my lab. Please let me know which environment you want to check.

        Attachments

          Activity

            People

            • Assignee:
              agalue Alejandro Galue
              Reporter:
              agalue Alejandro Galue
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: