Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13007

Prometheus Collector attempting to persist non-integer values to counters

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 27.0.0
    • Fix Version/s: 27.0.3
    • Component/s: Data Collection
    • Security Level: Default (Default Security Scheme)
    • Labels:
      None
    • Sprint:
      Horizon 2020 - Nov 11-Nov 24, Horizon 2020 - Nov 24-Dec 9

      Description

      RRD files initially created correctly but we only get NaNs (please see also attached rrdtool dump file total_idle.xml).
      This behaviour only seen with prometheus data with two labels - for example:

      windows_cpu_time_total{core="0,0",mode="idle"}

      Exporter data with max of one label store values correctly.

      tested with windows_exporter: https://github.com/prometheus-community/windows_exporter installed on Windows Server 2016

      to reproduce:

      • install windows_exporter on a windows node
      • please use attached datacollection configs and foreign-source configs to
      • add the windows node with installed prometheus exporter to a requisition and synchronize
      • wait some time, open node Resource Graphs and takes a look in the graph data

      example data from windows_exporter:

      windows_terminal_services_local_session_count{session="active"} 0
      windows_terminal_services_local_session_count{session="inactive"} 3
      windows_terminal_services_local_session_count{session="total"} 3
      windows_logical_disk_free_bytes{volume="C:"} 9.302966272e+09
      windows_logical_disk_free_bytes{volume="HarddiskVolume1"} 1.54140672e+08
      windows_logical_disk_size_bytes{volume="C:"} 4.2422239232e+10
      windows_logical_disk_size_bytes{volume="HarddiskVolume1"} 5.23239424e+08
      windows_cpu_time_total{core="0,0",mode="dpc"} 5.984375
      windows_cpu_time_total{core="0,0",mode="idle"} 248335.359375
      windows_cpu_time_total{core="0,0",mode="interrupt"} 15.25
      windows_cpu_time_total{core="0,0",mode="privileged"} 2093
      windows_cpu_time_total{core="0,0",mode="user"} 7767.921875
      windows_cpu_time_total{core="0,1",mode="dpc"} 46.03125
      windows_cpu_time_total{core="0,1",mode="idle"} 250622.234375
      windows_cpu_time_total{core="0,1",mode="interrupt"} 119.84375
      windows_cpu_time_total{core="0,1",mode="privileged"} 2224.484375
      windows_cpu_time_total{core="0,1",mode="user"} 5349.40625

      in collectd debug logs no errors found:

      2020-11-16 09:34:08,475 DEBUG [Collectd-Thread-9-of-150] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[total_idle:250299.53125]
      2020-11-16 09:34:08,476 DEBUG [Collectd-Thread-9-of-150] o.o.n.c.a.AbstractPersister: Persisting Attribute[total_idle:250299.53125]
      2020-11-16 09:34:08,476 DEBUG [Collectd-Thread-9-of-150] o.o.n.c.a.AbstractPersister: Storing attribute Attribute[total_idle:250299.53125]
      2020-11-16 09:34:08,476 INFO [Collectd-Thread-9-of-150] o.o.n.r.RrdMetaDataUtils: createMetaDataFile: creating meta data file /var/lib/opennms/rrd/snmp/fs/01-Test/1605513515011/winExporterCPU/0_0/total_idle.meta with values '{GROUP=windows_exporter_cpu_time, total_idle=total_idle}'
      2020-11-16 09:34:08,476 DEBUG [Collectd-Thread-9-of-150] o.o.n.r.r.MultithreadedJniRrdStrategy: createDefinition: filename [/var/lib/opennms/rrd/snmp/fs/01-Test/1605513515011/winExporterCPU/0_0/total_idle.rrd] already exists returning null as definition
      2020-11-16 09:34:08,476 DEBUG [Collectd-Thread-9-of-150] o.o.n.r.r.MultithreadedJniRrdStrategy: createRRD: skipping RRD file
      2020-11-16 09:34:08,476 INFO [Collectd-Thread-9-of-150] o.o.n.c.p.r.RrdPersistOperationBuilder: updateRRD: updating RRD file /var/lib/opennms/rrd/snmp/fs/01-Test/1605513515011/winExporterCPU/0_0/total_idle.rrd with values '1605515648:250299.53125'
      2020-11-16 09:34:08,476 DEBUG [Collectd-Thread-9-of-150] o.o.n.c.p.r.RrdPersistOperationBuilder: updateRRD: RRD update command completed.

       the rrd file exists and gets updates

      ls -al /var/lib/opennms/rrd/snmp/fs/01-Test/1605513515011/winExporterCPU/0_0/total_idle.rrd
      rw-rr- 1 root root 38232 Nov 16 09:34 /var/lib/opennms/rrd/snmp/fs/01-Test/1605513515011/winExporterCPU/0_0/total_idle.rrd

      and in graph data from Resource Graphs:

      Mon Nov 16 09:40:00 2020 NaN
      Mon Nov 16 09:35:00 2020 NaN
      Mon Nov 16 09:30:00 2020 NaN
      Mon Nov 16 09:25:00 2020 NaN
      Mon Nov 16 09:20:00 2020 NaN
      Mon Nov 16 09:15:00 2020 NaN
      Mon Nov 16 09:10:00 2020 NaN
      Mon Nov 16 09:05:00 2020 NaN

      please see also attached rrdtool dump

       

       

        Attachments

        1. opennms-config.zip
          5 kB
        2. total_idle.xml
          323 kB
        3. collectd.log
          84 kB
        4. Screenshot_20201118_133715.png
          Screenshot_20201118_133715.png
          35 kB

          Activity

            People

            Assignee:
            dino2gnt Dino Yancey
            Reporter:
            mlaercher Martin Laercher
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration