Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-9361

Thresholding on WSMan datasources appears broken

    Details

    • Sprint:
      Horizon - May 17th, Horizon - May 24th, Horizon - May 31st

      Description

      Thresholding on wsman resources appears to get confused about which instance is which, causing thresholds to flap or reset each other when processing thresholds for different datasources, e.g.

      2017-05-18 11:08:40,184 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeMBytes : {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158]
      2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeMBytes], value= 3797.0
      2017-05-18 11:08:40,184 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158], evaluating
      2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 3797.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]}
      *2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEvaluatorHighLow: evaluate: low threshold exceeded, count=1*
      2017-05-18 11:08:40,184 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeSpace : {evaluator=low, dsName=wmiLDPctFreeSpace, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeSpace, value=2.0, rearm=3.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158]
      2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeSpace], value= 5.0
      2017-05-18 11:08:40,184 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158], evaluating
      2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 5.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeSpace, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeSpace, value=2.0, rearm=3.0, trigger=2}]}
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.a.AttributeGroup: Visiting Group AttrGroupType[name=windows-os-wmi-LogicalDisk, ifType=all] for 0@1158
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDName:D:]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctFreeMBytes:436894.0]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: hasThresholds: wmiLogicalDisk@wmiLDPctFreeMBytes? true
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingVisitor: visitAttribute: storing value 436894.0 for attribute named wmiLDPctFreeMBytes
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctDiskRdTime:0.0]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctDiskWrTime:0.0]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctFreeSpace:94.0]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: hasThresholds: wmiLogicalDisk@wmiLDPctFreeSpace? true
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingVisitor: visitAttribute: storing value 94.0 for attribute named wmiLDPctFreeSpace
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDSplitIOsPerSec:0.0]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Applying thresholds on Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158] using 2 attributes.
      2017-05-18 11:08:40,185 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeMBytes : {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158]
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeMBytes], value= 436894.0
      2017-05-18 11:08:40,185 INFO  [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158], evaluating
      2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 436894.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]}
      *2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEvaluatorHighLow: evaluate: resetting low threshold count to 0, because the current value indicates that the in-progress threshold has been rearmed, but it doesn't triggered yet.*
      
      ResourceType
          <resourceType name="wmiLogicalDisk" label="(wsman) Logical Storage"
                        resourceLabel="Logical Disk (${wmiLDName})">
            <persistenceSelectorStrategy class="org.opennms.netmgt.collection.support.PersistAllSelectorStrategy"/>
            <storageStrategy class="org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy">
              <parameter key="sibling-column-name" value="wmiLDName" />
                      <parameter key="replace-first" value="s/^\d+\s//" />
                      <parameter key="replace-all" value="s/\s//" />
                      <parameter key="replace-all" value="s/:/_/" />
            </storageStrategy>
          </resourceType>
      
      collection group
          <group name="windows-os-wmi-LogicalDisk"
                 resource-uri="http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_PerfFormattedData_PerfDisk_LogicalDisk"
                 resource-type="wmiLogicalDisk">
            <!-- Label for this LogicalDisk instance -->
            <attrib name="Name" alias="wmiLDName" type="string"/>
      
            <!-- Unallocated space on the disk drive in megabytes -->
            <attrib name="FreeMegabytes" alias="wmiLDPctFreeMBytes" type="Gauge"/>
      
            <!-- Percentage of elapsed time that the selected disk drive is busy servicing read requests -->
            <attrib name="PercentDiskReadTime" alias="wmiLDPctDiskRdTime" type="Gauge"/>
      
            <!-- Percentage of elapsed time that the selected disk drive is busy servicing write requests -->
            <attrib name="PercentDiskWriteTime" alias="wmiLDPctDiskWrTime" type="Gauge"/>
      
            <!-- Ratio of the free space available on the logical disk unit to the total usable space -->
            <attrib name="PercentFreeSpace" alias="wmiLDPctFreeSpace" type="Gauge"/>
      
            <!-- Rate at which I/Os to the disk were split into multiple I/Os -->
            <attrib name="SplitIOPerSec" alias="wmiLDSplitIOsPerSec" type="Gauge"/>
          </group>
      
      Threshold group
          <group name="Ws-Man Windows Services" rrdRepository="/opt/opennms/share/rrd/snmp/">
              <threshold
                  description="Trigger an alarm when free space on a Windows Ws-Man volume is less than 10GB"
                  type="low" ds-type="wmiLogicalDisk" value="10240.0"
                  rearm="12288.0" trigger="2" ds-label="wmiLDName"
                  triggeredUEI="uei.opennms.org/threshold/lowThresholdExceededWSManLogStorage"
                  rearmedUEI="uei.opennms.org/threshold/lowThresholdRearmedWSManLogStorage"
                  filterOperator="or" ds-name="wmiLDPctFreeMBytes"/>
              <threshold
                  description="Trigger an alarm when free space on a Windows Ws-Man volume is less than 2%"
                  type="low" ds-type="wmiLogicalDisk" value="2.0" rearm="3.0"
                  trigger="2" ds-label="wmiLDName"
                  triggeredUEI="uei.opennms.org/threshold/lowThresholdExceededWSManLogStorage"
                  rearmedUEI="uei.opennms.org/threshold/lowThresholdRearmedWSManLogStorage"
                  filterOperator="or" ds-name="wmiLDPctFreeSpace"/>
          </group>
      

      Datacollection happens as expected, jrbs are populated and graphs work as expected. Thresholds never fire, or fire and immediately rearm when processing the next datasource.

      Not sure what else to check. No other datacollection mechanisms (jdbc, snmp, etc) exhibit this behavior.

        Attachments

          Activity

            People

            • Assignee:
              j-white Jesse White
              Reporter:
              dino2gnt Dino Yancey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: