Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-10831

Minion stops sending flow data into Kafka

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 25.0.0
    • Fix Version/s: None
    • Component/s: Minion
    • Security Level: Default (Default Security Scheme)
    • Labels:
    • Environment:
      opennms-minion-container.noarch 25.0.0-0.20190228.onms.develop.1644

      Description

      After some unknown interval, minion fails to send data into Kafka
      Below are excerpts from the logs which fill up almost instantly (see zgrep below)

      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-15 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-1 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-7 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-15 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-1 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,282 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-7 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,272 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-9 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      2019-04-01T17:42:55,272 | WARN  | OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-12 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      
      
      # zgrep Kafka * | sed 's/^[^\s]*\s//' | sort | uniq -cd | sort -n        
         2759 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-10 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
         2813 AggregatorFlush-Telemetry-Netflow-5 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
         2853 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-6 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        27432 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-15 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        27490 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-12 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        27878 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-16 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        28051 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-9 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        28497 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-11 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        28639 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-3 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        28864 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-14 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        28987 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-2 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        29237 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-8 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        29391 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-1 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        29597 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-13 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        29875 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-7 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        29877 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-5 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
        30124 OpenNMS.Sink.AsyncDispatcher.Telemetry-Netflow-5-Thread-4 | KafkaRemoteMessageDispatcherFactory | 249 - org.opennms.core.ipc.sink.kafka.client - 25.0.0.SNAPSHOT | Timeout occured while sending message to topic OpenNMS.Sink.Telemetry-Netflow-5, it will be attempted again.
      

        Attachments

          Activity

            People

            • Assignee:
              cgorantla Chandra Gorantla
              Reporter:
              sean Sean Torres
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: