SNMPPoller is the cause of loss of snmpinterfaces during the re-import / synchronization Provision Groups.
Description
Something is wrong with SNMPoller.
When you do re-import/synchronize Provision Groups which have a node with interfaces in the snmppoll="P" status (in the table snmpinterface), you will lose those interfaces for which the next scheduled scan of snmppoller occurred before completion of the phase of import node - the Node Scan Phase. Because SNMPoller writes in the field snmplastcapsdpoll old value (the time of previous import/node scan phase) when it updated snmplastsnmppoll value (or snmpif(admin|oper)status, or snmppoll), and provisiond, in turn, when complete the import node entity removes from the table snmpinterface all node interfaces wich snmplastcapsdpoll over time on the current node scan.
The workaround - delete from snmpinterface where snmppoll="P"; - before synchronize Provision Groups, or to stop SNMPoller before synchronize Provision Groups, or do not use SNMPoller in general.
^^by SNMPPoller when it update snmplastsnmppoll field during scheduled interface scan....then...again by provisiond
2010-07-29 14:56:02 EEST LOG: execute <unnamed>: delete from snmpInterface where nodeId=$1 and (snmpLastCapsdPoll is null or snmpLastCapsdPoll<$2) 2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '17', $2 = '2010-07-29 14:48:52.423+03' 2010-07-29 14:56:02 EEST LOG: execute <unnamed>: INSERT into events (eventID, eventUei, nodeID, eventTime, eventHost, ipAddr, eventDpName, eventSnmpHost, serviceID, eventSnmp, eventParms, eventCreateTime, eventDescr, eventLoggroup, eventLogmsg, eventLog, eventDisplay, eventSeverity, eventPathOutage, eventCorrelation, eventSuppressedCount, eventOperInstruct, eventAutoAction, eventOperAction, eventOperActionMenuText, eventNotification, eventTticket, eventTticketState, eventForward, eventMouseOverText, eventAckUser, eventAckTime, eventSource,ifIndex) values($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34) 2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '28668', $2 = 'uei.opennms.org/nodes/reinitializePrimarySnmpInterface', $3 = '17', $4 = '2010-07-29 14:56:02+03', $5 = NULL, $6 = '217.20.160.43', $7 = 'undefined', $8 = NULL, $9 = NULL, $10 = NULL, $11 = NULL, $12 = '2010-07-29 14:56:02.056+03', $13 = ' <p>A change in configuration on this node has been detected and the SNMP data collection mechanism is being triggered to refresh its required profile of the remote node.</p>', $14 = NULL, $15 = 'SNMP information on 217.20.160.43 is being refreshed for data collection purposes.', $16 = 'Y', $17 = 'Y', $18 = '4', $19 = NULL, $20 = NULL, $21 = NULL, $22 = NULL, $23 = NULL, $24 = NULL, $25 = NULL, $26 = NULL, $27 = NULL, $28 = NULL, $29 = NULL, $30 = NULL, $31 = NULL, $32 = NULL, $33 = 'Provisiond', $34 = NULL 2010-07-29 14:56:02 EEST LOG: execute <unnamed>: INSERT into events (eventID, eventUei, nodeID, eventTime, eventHost, ipAddr, eventDpName, eventSnmpHost, serviceID, eventSnmp, eventParms, eventCreateTime, eventDescr, eventLoggroup, eventLogmsg, eventLog, eventDisplay, eventSeverity, eventPathOutage, eventCorrelation, eventSuppressedCount, eventOperInstruct, eventAutoAction, eventOperAction, eventOperActionMenuText, eventNotification, eventTticket, eventTticketState, eventForward, eventMouseOverText, eventAckUser, eventAckTime, eventSource,ifIndex) values($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34) 2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '28669', $2 = 'uei.opennms.org/internal/provisiond/nodeScanCompleted', $3 = '17', $4 = '2010-07-29 14:56:02+03', $5 = NULL, $6 = NULL, $7 = 'undefined', $8 = NULL, $9 = NULL, $10 = NULL, $11 = 'foreignSource=BN(string,text);foreignId=1275309817574(string,text)', $12 = '2010-07-29 14:56:02.059+03', $13 = 'A message from the Provisiond NodeScan lifecycle that a NodeScan has completed: <p>The Node with Id: 17; ForeignSource: BN; ForeignId:1275309817574 has completed.</p> Typically the result of a request of an import request or a scheduled/user forced rescan.', $14 = NULL, $15 = ' <p>The Node with Id: 17; ForeignSource: BN; ForeignId:1275309817574 has completed.</p>', $16 = 'Y', $17 = 'Y', $18 = '3', $19 = NULL, $20 = NULL, $21 = NULL, $22 = NULL, $23 = NULL, $24 = NULL, $25 = NULL, $26 = NULL, $27 = NULL, $28 = NULL, $29 = NULL, $30 = NULL, $31 = NULL, $32 = NULL, $33 = 'Provisiond', $34 = NULL
^^^by provisiond - delete from the table all interfaces wich have snmplastcapsdpoll time over time on the current node scan.
Resalt we are lost some snmpinterfaces.
P.S2.: Another trouble with snmppoller, this is that when a node on which it monitors the interfaces is not available, then snmpif(admin|oper)status in the table did not change (that is, for ex., was 1(Up), and still is). What is not clear and not very convenient, for example, when used in automations. It would be nice if in such cases, determines the status and generate an appropriate event, such as unknown(4) and interface(Oper|Admin)Unknown.
Environment
Operating System: All
Platform: All
Acceptance / Success Criteria
None
Lucidchart Diagrams
Activity
Show:
Antonio Russo August 23, 2011 at 3:54 PM
fc5847b..36e06c1 1.10 -> 1.10
Antonio Russo August 23, 2011 at 3:51 PM
Fixed [1.10 53e71eb]
The update now update only ifadmin status, if oper status and lastSnmpPoll
Benjamin Reed August 22, 2011 at 8:55 AM
Probably best to fix in 1.10 first, and then we can see about backporting it after it goes through a bamboo cycle or two.
Eric W Abrahamsen August 22, 2011 at 8:23 AM
I would suggest fixing as soon as you can, this bug prevents me from adding nodes because it messes up data collection for anything filtered in the SNMP interface poller. My temporary fix is to disable the SNMP interface poller, add the nodes, and then re-enable. Thanks for looking into this!
Antonio Russo August 22, 2011 at 4:07 AM
Should be an easy fix. thanks for this good comments. I would fix this in 1.8 or 1.10?
Something is wrong with SNMPoller.
When you do re-import/synchronize Provision Groups which have a node with interfaces in the snmppoll="P" status (in the table snmpinterface), you will lose those interfaces for which the next scheduled scan of snmppoller occurred before completion of the phase of import node - the Node Scan Phase. Because SNMPoller writes in the field snmplastcapsdpoll old value (the time of previous import/node scan phase) when it updated snmplastsnmppoll value (or snmpif(admin|oper)status, or snmppoll), and provisiond, in turn, when complete the import node entity removes from the table snmpinterface all node interfaces wich snmplastcapsdpoll over time on the current node scan.
The workaround - delete from snmpinterface where snmppoll="P"; - before synchronize Provision Groups,
or to stop SNMPoller before synchronize Provision Groups,
or do not use SNMPoller in general.
P.S.:
2010-07-29 14:49:17 EEST LOG: execute <unnamed>: update snmpInterface set snmpCollect=$1, snmpIfAdminStatus=$2, snmpIfAlias=$
3, snmpIfDescr=$4, snmpIfIndex=$5, snmpIfName=$6, snmpIfOperStatus=$7, snmpIfSpeed=$8, snmpIfType=$9, ipAddr=$10, snmpLastCaps
dPoll=$11, snmpLastSnmpPoll=$12, snmpIpAdEntNetMask=$13, nodeId=$14, snmpPhysAddr=$15, snmpPoll=$16 where id=$17
2010-07-29 14:49:17 EEST DETAIL: parameters: $1 = 'C', $2 = '1', $3 = 'Nikolaev(hostn-new)_STM4', $4 = 'so-1/3/0', $5 = '77', $6 = 'so-1/3/0', $7 = '1', $8 = '622000000', $9 = '39', $10 = '0.0.0.0', $11 = '2010-07-29 14:48:52.423+03', $12 = '2010-07-29 14:47:33.745+03', $13 = NULL, $14 = '17', $15 = NULL, $16 = 'P', $17 = '15224'
^^^provisiond by "INFO [writeExecutor-7] DefaultProvisionService: Updating SnmpInterface" during Node Scan, then....
2010-07-29 14:52:34 EEST LOG: execute <unnamed>: update snmpInterface set snmpCollect=$1, snmpIfAdminStatus=$2, snmpIfAlias=$
3, snmpIfDescr=$4, snmpIfIndex=$5, snmpIfName=$6, snmpIfOperStatus=$7, snmpIfSpeed=$8, snmpIfType=$9, ipAddr=$10, snmpLastCaps
dPoll=$11, snmpLastSnmpPoll=$12, snmpIpAdEntNetMask=$13, nodeId=$14, snmpPhysAddr=$15, snmpPoll=$16 where id=$17
2010-07-29 14:52:34 EEST DETAIL: parameters: $1 = 'C', $2 = '1', $3 = 'Nikolaev(hostn-new)_STM4', $4 = 'so-1/3/0', $5 = '77', $6 = 'so-1/3/0', $7 = '1', $8 = '622000000', $9 = '39', $10 = '0.0.0.0', $11 = '2010-07-29 14:10:33.112+03', $12 = '2010-07-29 14:52:34.372+03', $13 = NULL, $14 = '17', $15 = NULL, $16 = 'P', $17 = '15224'
^^by SNMPPoller when it update snmplastsnmppoll field during scheduled interface scan....then...again by provisiond
2010-07-29 14:56:02 EEST LOG: execute <unnamed>: delete from snmpInterface where nodeId=$1 and (snmpLastCapsdPoll is null or snmpLastCapsdPoll<$2)
2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '17', $2 = '2010-07-29 14:48:52.423+03'
2010-07-29 14:56:02 EEST LOG: execute <unnamed>: INSERT into events (eventID, eventUei, nodeID, eventTime, eventHost, ipAddr, eventDpName, eventSnmpHost, serviceID, eventSnmp, eventParms, eventCreateTime, eventDescr, eventLoggroup, eventLogmsg, eventLog, eventDisplay, eventSeverity, eventPathOutage, eventCorrelation, eventSuppressedCount, eventOperInstruct, eventAutoAction, eventOperAction, eventOperActionMenuText, eventNotification, eventTticket, eventTticketState, eventForward, eventMouseOverText, eventAckUser, eventAckTime, eventSource,ifIndex) values($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34)
2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '28668', $2 = 'uei.opennms.org/nodes/reinitializePrimarySnmpInterface', $3 = '17', $4 = '2010-07-29 14:56:02+03', $5 = NULL, $6 = '217.20.160.43', $7 = 'undefined', $8 = NULL, $9 = NULL, $10 = NULL, $11 = NULL, $12 = '2010-07-29 14:56:02.056+03', $13 = ' <p>A change in configuration on this node has been detected and the SNMP data collection mechanism is being triggered to refresh its required profile of the remote node.</p>', $14 = NULL, $15 = 'SNMP information on 217.20.160.43 is being refreshed for data collection purposes.', $16 = 'Y', $17 = 'Y', $18 = '4', $19 = NULL, $20 = NULL, $21 = NULL, $22 = NULL, $23 = NULL, $24 = NULL, $25 = NULL, $26 = NULL, $27 = NULL, $28 = NULL, $29 = NULL, $30 = NULL, $31 = NULL, $32 = NULL, $33 = 'Provisiond', $34 = NULL
2010-07-29 14:56:02 EEST LOG: execute <unnamed>: INSERT into events (eventID, eventUei, nodeID, eventTime, eventHost, ipAddr, eventDpName, eventSnmpHost, serviceID, eventSnmp, eventParms, eventCreateTime, eventDescr, eventLoggroup, eventLogmsg, eventLog, eventDisplay, eventSeverity, eventPathOutage, eventCorrelation, eventSuppressedCount, eventOperInstruct, eventAutoAction, eventOperAction, eventOperActionMenuText, eventNotification, eventTticket, eventTticketState, eventForward, eventMouseOverText, eventAckUser, eventAckTime, eventSource,ifIndex) values($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34)
2010-07-29 14:56:02 EEST DETAIL: parameters: $1 = '28669', $2 = 'uei.opennms.org/internal/provisiond/nodeScanCompleted', $3 = '17', $4 = '2010-07-29 14:56:02+03', $5 = NULL, $6 = NULL, $7 = 'undefined', $8 = NULL, $9 = NULL, $10 = NULL, $11 = 'foreignSource=BN(string,text);foreignId=1275309817574(string,text)', $12 = '2010-07-29 14:56:02.059+03', $13 = 'A message from the Provisiond NodeScan lifecycle that a NodeScan has completed: <p>The Node with Id: 17; ForeignSource: BN; ForeignId:1275309817574 has completed.</p> Typically the result of a request of an import request or a scheduled/user forced rescan.', $14 = NULL, $15 = ' <p>The Node with Id: 17; ForeignSource: BN; ForeignId:1275309817574 has completed.</p>', $16 = 'Y', $17 = 'Y', $18 = '3', $19 = NULL, $20 = NULL, $21 = NULL, $22 = NULL, $23 = NULL, $24 = NULL, $25 = NULL, $26 = NULL, $27 = NULL, $28 = NULL, $29 = NULL, $30 = NULL, $31 = NULL, $32 = NULL, $33 = 'Provisiond', $34 = NULL
^^^by provisiond - delete from the table all interfaces wich have snmplastcapsdpoll time over time on the current node scan.
Resalt we are lost some snmpinterfaces.
P.S2.: Another trouble with snmppoller, this is that when a node on which it monitors the interfaces is not available, then snmpif(admin|oper)status in the table did not change (that is, for ex., was 1(Up), and still is). What is not clear and not very convenient, for example, when used in automations. It would be nice if in such cases, determines the status and generate an appropriate event, such as unknown(4) and interface(Oper|Admin)Unknown.