Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
Meridian-2016.1.4, 19.0.0, 19.0.1, 19.1.0
-
Security Level: Default (Default Security Scheme)
-
None
-
Horizon - May 17th, Horizon - May 24th, Horizon - May 31st
Description
Thresholding on wsman resources appears to get confused about which instance is which, causing thresholds to flap or reset each other when processing thresholds for different datasources, e.g.
2017-05-18 11:08:40,184 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeMBytes : {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158] 2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeMBytes], value= 3797.0 2017-05-18 11:08:40,184 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158], evaluating 2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 3797.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} *2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEvaluatorHighLow: evaluate: low threshold exceeded, count=1* 2017-05-18 11:08:40,184 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeSpace : {evaluator=low, dsName=wmiLDPctFreeSpace, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeSpace, value=2.0, rearm=3.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158] 2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeSpace], value= 5.0 2017-05-18 11:08:40,184 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@9520c7d, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@6cca9b1e]/Node[1158], evaluating 2017-05-18 11:08:40,184 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 5.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeSpace, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeSpace, value=2.0, rearm=3.0, trigger=2}]} 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.a.AttributeGroup: Visiting Group AttrGroupType[name=windows-os-wmi-LogicalDisk, ifType=all] for 0@1158 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDName:D:] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctFreeMBytes:436894.0] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: hasThresholds: wmiLogicalDisk@wmiLDPctFreeMBytes? true 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingVisitor: visitAttribute: storing value 436894.0 for attribute named wmiLDPctFreeMBytes 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctDiskRdTime:0.0] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctDiskWrTime:0.0] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDPctFreeSpace:94.0] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: hasThresholds: wmiLogicalDisk@wmiLDPctFreeSpace? true 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingVisitor: visitAttribute: storing value 94.0 for attribute named wmiLDPctFreeSpace 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.c.s.AbstractCollectionAttribute: Visiting attribute Attribute[wmiLDSplitIOsPerSec:0.0] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Applying thresholds on Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158] using 2 attributes. 2017-05-18 11:08:40,185 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: Processing threshold wmiLDPctFreeMBytes : {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} on resource Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158] 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.CollectionResourceWrapper: getAttributeValue: id=node[1158].resourceType[wmiLogicalDisk].instance[null].metric[wmiLDPctFreeMBytes], value= 436894.0 2017-05-18 11:08:40,185 INFO [Collectd-Thread-11-of-75] o.o.n.t.ThresholdingSet: applyThresholds: All attributes found for Resource[GenericTypeResource[node=NodeLevelResource[nodeId=1158], resourceType=ResourceType [name=wmiLogicalDisk, label=(wsman) Logical Storage, resourceLabel=Logical Disk (${wmiLDName}), persistenceSelectorStrategy=PersistenceSelectorStrategy [class=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy, parameters=[]], storageStrategy=StorageStrategy [class=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy, parameters=[Parameter [key=sibling-column-name, value=wmiLDName], Parameter [key=replace-first, value=s/^\d+\s//], Parameter [key=replace-all, value=s/\s//], Parameter [key=replace-all, value=s/:/_/]]]],storageStrategy=org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy@2934d8ac, persistenceSelectorStrategy=org.opennms.netmgt.collection.support.PersistAllSelectorStrategy@71c3a40]/Node[1158], evaluating 2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEntity: evaluate: value= 436894.0 against threshold: {evaluator=low, dsName=wmiLDPctFreeMBytes, dsType=wmiLogicalDisk, evaluators=[{ds=wmiLDPctFreeMBytes, value=10240.0, rearm=12288.0, trigger=2}]} *2017-05-18 11:08:40,185 DEBUG [Collectd-Thread-11-of-75] o.o.n.t.ThresholdEvaluatorHighLow: evaluate: resetting low threshold count to 0, because the current value indicates that the in-progress threshold has been rearmed, but it doesn't triggered yet.*
ResourceType
<resourceType name="wmiLogicalDisk" label="(wsman) Logical Storage" resourceLabel="Logical Disk (${wmiLDName})"> <persistenceSelectorStrategy class="org.opennms.netmgt.collection.support.PersistAllSelectorStrategy"/> <storageStrategy class="org.opennms.netmgt.dao.support.SiblingColumnStorageStrategy"> <parameter key="sibling-column-name" value="wmiLDName" /> <parameter key="replace-first" value="s/^\d+\s//" /> <parameter key="replace-all" value="s/\s//" /> <parameter key="replace-all" value="s/:/_/" /> </storageStrategy> </resourceType>
collection group
<group name="windows-os-wmi-LogicalDisk" resource-uri="http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_PerfFormattedData_PerfDisk_LogicalDisk" resource-type="wmiLogicalDisk"> <!-- Label for this LogicalDisk instance --> <attrib name="Name" alias="wmiLDName" type="string"/> <!-- Unallocated space on the disk drive in megabytes --> <attrib name="FreeMegabytes" alias="wmiLDPctFreeMBytes" type="Gauge"/> <!-- Percentage of elapsed time that the selected disk drive is busy servicing read requests --> <attrib name="PercentDiskReadTime" alias="wmiLDPctDiskRdTime" type="Gauge"/> <!-- Percentage of elapsed time that the selected disk drive is busy servicing write requests --> <attrib name="PercentDiskWriteTime" alias="wmiLDPctDiskWrTime" type="Gauge"/> <!-- Ratio of the free space available on the logical disk unit to the total usable space --> <attrib name="PercentFreeSpace" alias="wmiLDPctFreeSpace" type="Gauge"/> <!-- Rate at which I/Os to the disk were split into multiple I/Os --> <attrib name="SplitIOPerSec" alias="wmiLDSplitIOsPerSec" type="Gauge"/> </group>
Threshold group
<group name="Ws-Man Windows Services" rrdRepository="/opt/opennms/share/rrd/snmp/"> <threshold description="Trigger an alarm when free space on a Windows Ws-Man volume is less than 10GB" type="low" ds-type="wmiLogicalDisk" value="10240.0" rearm="12288.0" trigger="2" ds-label="wmiLDName" triggeredUEI="uei.opennms.org/threshold/lowThresholdExceededWSManLogStorage" rearmedUEI="uei.opennms.org/threshold/lowThresholdRearmedWSManLogStorage" filterOperator="or" ds-name="wmiLDPctFreeMBytes"/> <threshold description="Trigger an alarm when free space on a Windows Ws-Man volume is less than 2%" type="low" ds-type="wmiLogicalDisk" value="2.0" rearm="3.0" trigger="2" ds-label="wmiLDName" triggeredUEI="uei.opennms.org/threshold/lowThresholdExceededWSManLogStorage" rearmedUEI="uei.opennms.org/threshold/lowThresholdRearmedWSManLogStorage" filterOperator="or" ds-name="wmiLDPctFreeSpace"/> </group>
Datacollection happens as expected, jrbs are populated and graphs work as expected. Thresholds never fire, or fire and immediately rearm when processing the next datasource.
Not sure what else to check. No other datacollection mechanisms (jdbc, snmp, etc) exhibit this behavior.