OpenNMS

Provisiond - simple TCP detector fails to detect services

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.8.1
  • Fix Version/s: 1.8.11, 1.9.7
  • Component/s: Provisioning/Discovery/Capsd/Provisiond
  • Security Level: Default (Default Security Scheme)
  • Labels:
    None
  • Environment:
    Operating System: Linux
    Platform: PC
  • Bugzilla Id:
    3996

Description

Debian Linux - 2.6.26-1-amd64 #1 SMP x86_64 GNU/Linux
OpenNMS 1.8.1 from from the ONMS deb repo

Provisiond fails to find open TCP ports defined using the simple TCP detector.

Stock installation, no config changes apart from a provisioning group and a custom foreign source "Printers".

In the foreign source for the provisoining group Printers I have an entry:

        <detector class="org.opennms.netmgt.provision.detector.simple.TcpDetector" name="IPP">
            <parameter value="*" key="banner"/>
            <parameter value="631" key="port"/>
            <parameter value="3000" key="timeout"/>
            <parameter value="2" key="retries"/>
        </detector>

I can then add a simple node definition to the provisioning group:

    <node node-label="print01" foreign-id="1279287949812" building="Printer">
        <interface status="1" snmp-primary="P" ip-addr="1.1.1.1" descr="main">
            <monitored-service service-name="SNMP"/>
            <monitored-service service-name="ICMP"/>
        </interface>
    </node>

Please note that I'm deliberately only defining ICMP and SNMP, so that the rest of the services are detected, not provisioned.

If I then import the provisioning group, it fails to detect the IPP service. In fact any service I define using the Simple TcpDetector seems to fail to be detected, but other such as HTTP are found.


As a sanity check, I nmap the printer from the same host that runs ONMS:


Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2010-07-16 15:26 BST
Interesting ports on print1.mydom.com (1.1.1.1):
Not shown: 1668 closed ports
PORT STATE SERVICE
25/tcp open smtp
80/tcp open http
443/tcp open https
515/tcp open printer
631/tcp open ipp
1521/tcp filtered oracle
1522/tcp filtered rna-lm
1523/tcp filtered cichild-lm
1524/tcp filtered ingreslock
1525/tcp filtered orasrv
8000/tcp open http-alt
9100/tcp open jetdirect

There we see TCP:631 is open, and I can also telnet/nc to it.

If I tcpdump the traffic during the provisioning/detection stage on teh ONMS server, I see nothing unusual. In fact I see some promising interaction, a three way handshake to start a connection on that port, followed by a four way handshake to terminate the connection at both ends. Clearly the detector is running:

14:46:48.195381 IP onms.mydom.com.52636 > print01.mydom.com.ipp: S 4218595545:4218595545(0) win 5840 <mss 1460,sackOK,timestamp 2352565653 0,nop,wscale 7>
14:46:48.196039 IP print01.mydom.com.ipp > onms.mydom.com.52636: S 449433445:449433445(0) ack 4218595546 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 39554513 2352565653>
14:46:48.196089 IP onms.mydom.com.52636 > print01.mydom.com.ipp: . ack 1 win 46 <nop,nop,timestamp 2352565653 39554513>
14:46:49.205409 IP onms.mydom.com.52636 > print01.mydom.com.ipp: F 1:1(0) ack 1 win 46 <nop,nop,timestamp 2352565906 39554513>
14:46:49.206088 IP print01.mydom.com.ipp > onms.mydom.com.52636: . ack 2 win 17376 <nop,nop,timestamp 39554574 2352565906>
14:46:49.206378 IP print01.mydom.com.ipp > onms.mydom.com.52636: F 1:1(0) ack 2 win 17376 <nop,nop,timestamp 39554574 2352565906>
14:46:49.206405 IP onms.mydom.com.52636 > print01.mydom.com.ipp: . ack 2 win 46 <nop,nop,timestamp 2352565906 39554574>


I've gone over the log thoroughly, and can't find much mention of the detection. All I see is:

2010-07-16 15:35:50,112 INFO [Provisiond-TaskScheduler] AsyncDetectorRunner: Attemping to detect service IPP on address 1.1.1.1

~~~ snip ~~~

2010-07-16 15:35:51,123 INFO [NioProcessor-316] IpInterfaceScan$1: Attempted to detect service IPP on address 1.1.1.1: false
2010-07-16 15:35:51,123 INFO [NioProcessor-316] TcpDetector: calling dispose on detector IPP

Where I've "snipped", there is no mention of IPP or port 631 at all, I see no other log entries that are relevant. However, if someone deems it necessary to resolve, I will post the whole thing..
BTW I am logging DEBUG messages for provisiond.


The fact that I've defined the service detector to retry 2 times, and I only see one connection, also makes me think that the detection is successful.

Something is wrong somewhere :)

I posted this on the mailing group, but one one else seemed to be able to help, so I'm raising this as a bug.

Please let me know if you require any further information.



Cheers,
Just

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Benjamin Reed added a comment -

Curious, why is banner set to "*"? I believe that will match only a banner that contains an asterisk. If you want a regular expression that matches anything, I think you want:

<parameter key="banner" value="~.*" />

Show
Benjamin Reed added a comment - Curious, why is banner set to "*"? I believe that will match only a banner that contains an asterisk. If you want a regular expression that matches anything, I think you want: <parameter key="banner" value="~.*" />
Hide
Permalink
jcat added a comment -

I always used a "*" with capsd, and it always used to work.

Anyway, I have tried the regex version as well, and it don't work either.
I just tried it again to make sure, no go.

Cheers,
Just

Show
jcat added a comment - I always used a "*" with capsd, and it always used to work. Anyway, I have tried the regex version as well, and it don't work either. I just tried it again to make sure, no go. Cheers, Just
Hide
Permalink
Preston Gallwas added a comment -

(In reply to comment #2)
> I always used a "*" with capsd, and it always used to work.
>
> Anyway, I have tried the regex version as well, and it don't work either.
> I just tried it again to make sure, no go.
>
>
> Cheers,
> Just
>

FWIW I'm seeing that services aren't being detected either with TCPDetector or Win32ServiceDetector (see my NMS-4047). When the 'default' TCPDetector failed to find Groupwise (which works with CAPSD detecting 1677) I made a custom TCPDetector with the "port" parameter set to 1677.

I did see

"2010-08-05 14:00:49,435 INFO [NioProcessor-970] IpInterfaceScan$1: Attempted to detect service Groupwise on address 10.1.1.3: false"

Telnet to that port shows a successful connection.

Show
Preston Gallwas added a comment - (In reply to comment #2) > I always used a "*" with capsd, and it always used to work. > > Anyway, I have tried the regex version as well, and it don't work either. > I just tried it again to make sure, no go. > > > Cheers, > Just > FWIW I'm seeing that services aren't being detected either with TCPDetector or Win32ServiceDetector (see my NMS-4047 ). When the 'default' TCPDetector failed to find Groupwise (which works with CAPSD detecting 1677) I made a custom TCPDetector with the "port" parameter set to 1677. I did see "2010-08-05 14:00:49,435 INFO [NioProcessor-970] IpInterfaceScan$1: Attempted to detect service Groupwise on address 10.1.1.3: false" Telnet to that port shows a successful connection.
Hide
Permalink
desloge@opennms.org added a comment -

I recently was working on a bug similar to this, where the TcpDetector wouldn't detect a TCP connection with expecting a banner back. It was found that the TcpDetector was always expecting a banner no matter what and if it didn't receive one it would fail. The regex that was used for testing for any banner was ".*".

It has been fixed and the latest behavior of the TcpDetector is as follows. If you specify a banner in your settings it will expect to receive a banner and will fail if the banner is not sent. Even if its setup to expect any banner. If you do not specify a banner in your setting then it will not expect a banner and will detect a connection if one is made. This was fixed on 8/25/2010 and is in trunk and it should make it into the next upcoming release.

Show
desloge@opennms.org added a comment - I recently was working on a bug similar to this, where the TcpDetector wouldn't detect a TCP connection with expecting a banner back. It was found that the TcpDetector was always expecting a banner no matter what and if it didn't receive one it would fail. The regex that was used for testing for any banner was ".*". It has been fixed and the latest behavior of the TcpDetector is as follows. If you specify a banner in your settings it will expect to receive a banner and will fail if the banner is not sent. Even if its setup to expect any banner. If you do not specify a banner in your setting then it will not expect a banner and will detect a connection if one is made. This was fixed on 8/25/2010 and is in trunk and it should make it into the next upcoming release.
Hide
Permalink
Seth Leger (community account) added a comment -

Marking this as fixed based on the changes Donald made in commit 8ea5040011e1dab3e0220a6b0ca63029735f6981.

Show
Seth Leger (community account) added a comment - Marking this as fixed based on the changes Donald made in commit 8ea5040011e1dab3e0220a6b0ca63029735f6981.
Hide
Permalink
Jason Lixfeld added a comment -

This appears to still be broken as of 1.8.9.

Without a banner specific, I get this in provisiond.log:

2011-02-22 22:15:58,325 DEBUG [scanExecutor-6] IpInterfaceScan: detectServices for 51 : 10.219.49.2: found 1 detectors
2011-02-22 22:15:58,430 INFO [Provisiond-TaskScheduler] AsyncDetectorRunner: Attemping to detect service Telnet on address 10.219.49.2
2011-02-22 22:15:58,510 DEBUG [NioProcessor-6] BaseDetectorHandler: Client Receiving: Response: ^X #'$!

2011-02-22 22:15:58,517 INFO [NioProcessor-6] IpInterfaceScan$1: Attempted to detect service Telnet on address 10.219.49.2: false

With a banner of .* specified, I get this in provisiond.log:

2011-02-22 22:20:47,996 DEBUG [scanExecutor-2] IpInterfaceScan: detectServices for 51 : 10.219.49.2: found 1 detectors
2011-02-22 22:20:47,996 INFO [Provisiond-TaskScheduler] AsyncDetectorRunner: Attemping to detect service Telnet on address 10.219.49.2
2011-02-22 22:20:48,003 DEBUG [NioProcessor-271] BaseDetectorHandler: Client Receiving: Response: ???? #'$!

2011-02-22 22:20:48,004 INFO [NioProcessor-271] IpInterfaceScan$1: Attempted to detect service Telnet on address 10.219.49.1: true

Show
Jason Lixfeld added a comment - This appears to still be broken as of 1.8.9. Without a banner specific, I get this in provisiond.log: 2011-02-22 22:15:58,325 DEBUG [scanExecutor-6] IpInterfaceScan: detectServices for 51 : 10.219.49.2: found 1 detectors 2011-02-22 22:15:58,430 INFO [Provisiond-TaskScheduler] AsyncDetectorRunner: Attemping to detect service Telnet on address 10.219.49.2 2011-02-22 22:15:58,510 DEBUG [NioProcessor-6] BaseDetectorHandler: Client Receiving: Response: ^X # ' $ ! 2011-02-22 22:15:58,517 INFO [NioProcessor-6] IpInterfaceScan$1: Attempted to detect service Telnet on address 10.219.49.2: false With a banner of .* specified, I get this in provisiond.log: 2011-02-22 22:20:47,996 DEBUG [scanExecutor-2] IpInterfaceScan: detectServices for 51 : 10.219.49.2: found 1 detectors 2011-02-22 22:20:47,996 INFO [Provisiond-TaskScheduler] AsyncDetectorRunner: Attemping to detect service Telnet on address 10.219.49.2 2011-02-22 22:20:48,003 DEBUG [NioProcessor-271] BaseDetectorHandler: Client Receiving: Response: ???? # ' $ ! 2011-02-22 22:20:48,004 INFO [NioProcessor-271] IpInterfaceScan$1: Attempted to detect service Telnet on address 10.219.49.1: true
Hide
Permalink
Benjamin Reed added a comment -

I think the regexes need to be anchored. Does it work with a banner of: ^.*$

?

Show
Benjamin Reed added a comment - I think the regexes need to be anchored. Does it work with a banner of: ^.*$ ?
Hide
Permalink
Ronny Trommer added a comment -

not detected

Show
Ronny Trommer added a comment - not detected
Hide
Permalink
Ronny Trommer added a comment -

Hi guys,

I found a very weird issue with TCP-Dector today, I guess the functionality is not exactly the same as in capsd. I've seen there are open TCP services detected and some are not detected. So far I can see the problem exist only if the server displays no connection banner.

I found an issue from jcat, which describes really good my current problem, but his is resolved.

http://issues.opennms.org/browse/NMS-4033

For reproduction, I found a good example service which is not hard to find MS RDP TCP/3389. To check if the port is open and reachable, I've used the
easy telnet connect with

http://issues.opennms.org/secure/attachment/11731/screenshot-2.jpg

telnet myserver 3389

Trying myserver...
Connected to myserver.
Escape character is '^]'.

no server banner and connection is established. Secondly verified from my opennms box with nmap

nmap -v -p 3389 myserver
3389/tcp open ms-term-serv

I've found a SSL IRC server which is detectable and has also no connection banner. Tried with the same
procedure with telnet and nmap

telnet myircserver 6697

nmap -v -p 6697
Trying myserver...
Connected to myircserver.
Escape character is '^]'.

6697/tcp open unknown

My detector configuration for the detectors:

<detector class="org.opennms.netmgt.provision.detector.simple.TcpDetector" name="TCP-RDP">
<parameter value="3389" key="port"/>
<parameter value="1000" key="timeout"/>
</detector>
<detector class="org.opennms.netmgt.provision.detector.simple.TcpDetector" name="IRC-6697">
<parameter value="6697" key="port"/>
<parameter value="1000" key="timeout"/>
</detector>
</detectors>

Result, the IRC server is detected, the RDP service is not detected. Debugging in provisiond with no error, even a false for the detected service. I've seen, hit the "synchronize" in the provsioning group, I can see with

netstat -n | grep 3389

an established connection for ~ 1 minute. Next step was tcpdump the connection to see what really happened. I've seen the behavior from the SSL IRC daemon is straight forward. OpenNMS establishes the connection, then the disconnection is normal. The service is detected. The tcpdump is shown in

http://issues.opennms.org/secure/attachment/11730/screenshot-1.jpg

The behavior on RDP is different, the connection is established and stays in this state
for ~1 minute and then a RST is sent from the RDP-Server.

http://issues.opennms.org/secure/attachment/11731/screenshot-2.jpg

OpenNMS does not detect the service and is not monitored. A really nasty side effect the thread for the connection is bound for 1 minute. Which means using a lot of TCP-detectors will bind a lot of threads in blocking-state until the server sends a RST.

I used OpenNMS stable 1.8.10. Before I'll open a resolved bug, it would be nice if somebody can verify this issue.

Show
Ronny Trommer added a comment - Hi guys, I found a very weird issue with TCP-Dector today, I guess the functionality is not exactly the same as in capsd. I've seen there are open TCP services detected and some are not detected. So far I can see the problem exist only if the server displays no connection banner. I found an issue from jcat, which describes really good my current problem, but his is resolved. http://issues.opennms.org/browse/NMS-4033 For reproduction, I found a good example service which is not hard to find MS RDP TCP/3389. To check if the port is open and reachable, I've used the easy telnet connect with http://issues.opennms.org/secure/attachment/11731/screenshot-2.jpg telnet myserver 3389 Trying myserver... Connected to myserver. Escape character is '^]'. no server banner and connection is established. Secondly verified from my opennms box with nmap nmap -v -p 3389 myserver 3389/tcp open ms-term-serv I've found a SSL IRC server which is detectable and has also no connection banner. Tried with the same procedure with telnet and nmap telnet myircserver 6697 nmap -v -p 6697 Trying myserver... Connected to myircserver. Escape character is '^]'. 6697/tcp open unknown My detector configuration for the detectors: <detector class="org.opennms.netmgt.provision.detector.simple.TcpDetector" name="TCP-RDP"> <parameter value="3389" key="port"/> <parameter value="1000" key="timeout"/> </detector> <detector class="org.opennms.netmgt.provision.detector.simple.TcpDetector" name="IRC-6697"> <parameter value="6697" key="port"/> <parameter value="1000" key="timeout"/> </detector> </detectors> Result, the IRC server is detected, the RDP service is not detected. Debugging in provisiond with no error, even a false for the detected service. I've seen, hit the "synchronize" in the provsioning group, I can see with netstat -n | grep 3389 an established connection for ~ 1 minute. Next step was tcpdump the connection to see what really happened. I've seen the behavior from the SSL IRC daemon is straight forward. OpenNMS establishes the connection, then the disconnection is normal. The service is detected. The tcpdump is shown in http://issues.opennms.org/secure/attachment/11730/screenshot-1.jpg The behavior on RDP is different, the connection is established and stays in this state for ~1 minute and then a RST is sent from the RDP-Server. http://issues.opennms.org/secure/attachment/11731/screenshot-2.jpg OpenNMS does not detect the service and is not monitored. A really nasty side effect the thread for the connection is bound for 1 minute. Which means using a lot of TCP-detectors will bind a lot of threads in blocking-state until the server sends a RST. I used OpenNMS stable 1.8.10. Before I'll open a resolved bug, it would be nice if somebody can verify this issue.
Hide
Permalink
Donald Desloge added a comment -

Turns out there must have been a bug in Mina, the underlying Asychronous NIO library we use. When waiting for the detector to detect a service it won't wait more than 4 - 5 seconds. It would always return false, if the TCP service you are trying to reach has a longer timeout than 5 seconds it would return false.

I've since then bumped up the Mina version and that seems to fix it. Changed version from 2.0.0-M2 to 2.0.2 . In your OpenNMS install, you can remove the old version and drop in the new version and see if that corrects the problem.

Show
Donald Desloge added a comment - Turns out there must have been a bug in Mina, the underlying Asychronous NIO library we use. When waiting for the detector to detect a service it won't wait more than 4 - 5 seconds. It would always return false, if the TCP service you are trying to reach has a longer timeout than 5 seconds it would return false. I've since then bumped up the Mina version and that seems to fix it. Changed version from 2.0.0-M2 to 2.0.2 . In your OpenNMS install, you can remove the old version and drop in the new version and see if that corrects the problem.
Hide
Permalink
Donald Desloge added a comment -

Updated Mina version, this fixes the issue.

Show
Donald Desloge added a comment - Updated Mina version, this fixes the issue.
Hide
Permalink
Donald Desloge added a comment -

Closing, it has been verified as fixed.

Show
Donald Desloge added a comment - Closing, it has been verified as fixed.

People

  • Assignee:
    Matt Brozowski
    Reporter:
    jcat
Vote (1)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: