OpenNMS

Intermittent UnknownHostException thrown during AsyncDetectorFileDescriptorLeakTest

Details

Description

While investigating the issues from NMS-4846, I am seeing VERY intermittent problems inside the AsyncDetectorFileDescriptorLeakTest where it occasionally gets an UnknownHostException when trying to bind to localhost. This problem only occurs about once every 20,000 times on my Fedora Linux VMware image. This causes the isServiceDetected() method to return a false negative. I think that the solution is to use our InetAddressUtils util functions to return a localhost InetAddress without throwing any exceptions.

Issue Links

blocks

Bug - A problem which impairs or prevents the functions of the product. NMS-4846 Provisiond leaks file handles, eventually causing "Too many open files" crashes

  • Blocker - Blocks development and/or testing work, production could not run.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Seth Leger added a comment -

Here is what the logs look like for the unit test when this occurs.

2012-04-18 10:02:28,100 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: Expected Rate: 0.200000 Actual Rate: 0.196683 Events Sent: 3190
2012-04-18 10:02:28,100 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: current loop: 3190
2012-04-18 10:02:28,101 INFO  [main] org.opennms.netmgt.provision.support.AsyncBasicDetector$2: Connection exception occurred java.net.UnknownHostException: delta: delta for service TCP, retrying attempt 1
2012-04-18 10:02:28,119 DEBUG [main] org.apache.mina.core.service.AbstractIoService: awaitTermination on (nio socket connector: managedSessionCount: 0) called by thread=[main]
2012-04-18 10:02:28,127 DEBUG [main] org.apache.mina.core.service.AbstractIoService: awaitTermination on (nio socket connector: managedSessionCount: 0) finished
2012-04-18 10:02:28,141 INFO  [main] org.opennms.netmgt.provision.support.AsyncBasicDetector$2: Service TCP detected false
2012-04-18 10:02:28,141 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: got value: false
Show
Seth Leger added a comment - Here is what the logs look like for the unit test when this occurs. 2012-04-18 10:02:28,100 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: Expected Rate: 0.200000 Actual Rate: 0.196683 Events Sent: 3190 2012-04-18 10:02:28,100 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: current loop: 3190 2012-04-18 10:02:28,101 INFO [main] org.opennms.netmgt.provision.support.AsyncBasicDetector$2: Connection exception occurred java.net.UnknownHostException: delta: delta for service TCP, retrying attempt 1 2012-04-18 10:02:28,119 DEBUG [main] org.apache.mina.core.service.AbstractIoService: awaitTermination on (nio socket connector: managedSessionCount: 0) called by thread=[main] 2012-04-18 10:02:28,127 DEBUG [main] org.apache.mina.core.service.AbstractIoService: awaitTermination on (nio socket connector: managedSessionCount: 0) finished 2012-04-18 10:02:28,141 INFO [main] org.opennms.netmgt.provision.support.AsyncBasicDetector$2: Service TCP detected false 2012-04-18 10:02:28,141 DEBUG [main] org.opennms.netmgt.provision.detector.AsyncDetectorFileDescriptorLeakTest: got value: false
Hide
Permalink
Seth Leger added a comment -

This problem was fixed in 1.10.3 by using a call to NioSocketConnector that specifies the localhost SocketAddress explicitly. To construct the SocketAddress, we can use an InetAddressUtils call that never uses DNS to look up the localhost interface (it just uses 127.0.0.1), thereby bypassing the intermittent problem.

Show
Seth Leger added a comment - This problem was fixed in 1.10.3 by using a call to NioSocketConnector that specifies the localhost SocketAddress explicitly. To construct the SocketAddress, we can use an InetAddressUtils call that never uses DNS to look up the localhost interface (it just uses 127.0.0.1), thereby bypassing the intermittent problem.

People

  • Assignee:
    Seth Leger
    Reporter:
    Seth Leger
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: