output.log reports java.net.ConnectException error while starting

Description

OpenNMS reports a java exception as below when starting with vanilla configs and vanilla DB.
If I disable provisiond in service-configuration.xml, the error disappears.

This a custom compile, not a debian package, however the compile precess reported no errors and it's the same process I use regularly.

Tue Mar 13 18:26:38 GMT 2012
begin ulimit settings:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 20480
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
end ulimit settings
Executing command: /opennms/current/bin/runjava -r – -Djava.endorsed.dirs=/opennms/current/lib/endorsed -Dopennms.home=/opennms/current -Dcom.sun.management.jmxremote.port=18980 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=128m -jar /opennms/current/lib/opennms_bootstrap.jar start
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib/jni/libjicmp.so. Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[DEBUG] System property 'opennms.library.jicmp6' not set. Attempting to load library using System.loadLibrary("jicmp6").
13-Mar-2012 18:27:25 com.noelios.restlet.http.StreamClientHelper start
INFO: Starting the HTTP client
13-Mar-2012 18:27:25 com.noelios.restlet.http.StreamClientCall sendRequest
WARNING: An error occured during the communication with the remote HTTP server.
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at com.noelios.restlet.http.StreamClientCall.createSocket(StreamClientCall.java:151)
at com.noelios.restlet.http.StreamClientCall.sendRequest(StreamClientCall.java:319)
at com.noelios.restlet.http.HttpClientConverter.commit(HttpClientConverter.java:418)

at com.noelios.restlet.http.HttpClientHelper.handle(HttpClientHelper.java:108)
at org.restlet.Client.handle(Client.java:158)
at org.restlet.Uniform.handle(Uniform.java:108)
at org.opennms.rancid.RWSClientApi.getMethodRWS(RWSClientApi.java:812)
at org.opennms.rancid.RWSClientApi.getInfo(RWSClientApi.java:210)
at org.opennms.rancid.RWSClientApi.getRWSResourceDeviceTypesPatternList(RWSClientApi.java:178)
at org.opennms.netmgt.provision.RancidProvisioningAdapter.getRancidCategories(RancidProvisioningAdapter.java:169)
at org.opennms.netmgt.provision.RancidProvisioningAdapter.afterPropertiesSet(RancidProvisioningAdapter.java:156)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:398)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1146)
at org.springframework.beans.factory.BeanFactoryUtils.beansOfTypeIncludingAncestors(BeanFactoryUtils.java:275)
at org.opennms.netmgt.provision.service.DefaultPluginRegistry.beansOfType(DefaultPluginRegistry.java:159)
at org.opennms.netmgt.provision.service.DefaultPluginRegistry.getAllPlugins(DefaultPluginRegistry.java:135)
at org.opennms.netmgt.provision.service.ProvisioningAdapterManager.afterPropertiesSet(ProvisioningAdapterManager.java:68)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1079)
at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:397)
at org.opennms.core.utils.BeanUtils.getBeanFactory(BeanUtils.java:51)
at org.opennms.core.utils.BeanUtils.getFactory(BeanUtils.java:93)
at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.init(AbstractSpringContextJmxServiceDaemon.java:97)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:324)
at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:247)
at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:245)
at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:116)
at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:190)
at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.opennms.bootstrap.Bootstrap$3.run(Bootstrap.java:346)
at java.lang.Thread.run(Thread.java:662)

13-Mar-2012 18:27:43 com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.opennms.web.rest
13-Mar-2012 18:27:44 com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.opennms.web.rest.ForeignSourceRestService
class org.opennms.web.rest.ScheduledOutagesRestService
class org.opennms.web.rest.NotificationRestService
class org.opennms.web.rest.OutageRestService
class org.opennms.web.rest.EventRestService
class org.opennms.web.rest.UserRestService
class org.opennms.web.rest.AlarmRestService
class org.opennms.web.rest.OnmsMapRestService
class org.opennms.web.rest.AcknowledgmentRestService
class org.opennms.web.rest.AlarmStatsRestService
class org.opennms.web.rest.NodeRestService
class org.opennms.web.rest.SnmpConfigRestService
class org.opennms.web.rest.AssetRecordResource
class org.opennms.web.rest.RemotePollerAvailabilityService
class org.opennms.web.rest.GroupRestService
class org.opennms.web.rest.RequisitionRestService
class org.opennms.web.rest.OnmsCategoryResource
13-Mar-2012 18:27:44 com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.opennms.web.rest.ValidatingMessageBodyReader
class org.opennms.web.rest.FormPropertiesReader
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, notificationRestService, of type org.opennms.web.rest.NotificationRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, snmpConfigRestService, of type org.opennms.web.rest.SnmpConfigRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans

INFO: Registering Spring bean, alarmStatsRestService, of type org.opennms.web.rest.AlarmStatsRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, remotePollerAvailabilityService, of type org.opennms.web.rest.RemotePollerAvailabilityService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, acknowledgmentRestService, of type org.opennms.web.rest.AcknowledgmentRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetRecordResource, of type org.opennms.web.rest.AssetRecordResource as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, outageRestService, of type org.opennms.web.rest.OutageRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, alarmRestService, of type org.opennms.web.rest.AlarmRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, nodeRestService, of type org.opennms.web.rest.NodeRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, requisitionRestService, of type org.opennms.web.rest.RequisitionRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, onmsMapRestService, of type org.opennms.web.rest.OnmsMapRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, foreignSourceRestService, of type org.opennms.web.rest.ForeignSourceRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, onmsCategoryResource, of type org.opennms.web.rest.OnmsCategoryResource as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, scheduledOutagesRestService, of type org.opennms.web.rest.ScheduledOutagesRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, groupRestService, of type org.opennms.web.rest.GroupRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, eventRestService, of type org.opennms.web.rest.EventRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userRestService, of type org.opennms.web.rest.UserRestService as a root resource class
13-Mar-2012 18:27:44 com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.5 01/14/2011 12:36 PM'
13-Mar-2012 18:27:45 de.novanic.eventservice.logger.DefaultServerLogger log
INFO: Server: Configuration changed - EventServiceConfiguration (Default Configuration)
Min.: 0ms; Max.: 20000ms; Timeout: 90000ms
WARNING: register() called but the service has already been m_registered!
WARNING: register() called but the service has already been m_registered!

Cheers,
Just

Environment

Debian Squeeze - Linux 3.2.0-0.bpo.1-amd64 (amd64) - 1.6.0_26 Sun Microsystems Inc.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

jcat March 27, 2012 at 7:28 AM

s/done/one/g

Cheers,
Just

jcat March 27, 2012 at 6:22 AM

Thanks for the responses.
Apologies for not responding myself - I've been on holiday.

I do have done question:
Given that I'm not doing a "-p full (or fulldir)" as far as I know - do the compile scripts need fixing up, or am I doing it incorrectly?

Cheers,
Just

Seth Leger March 20, 2012 at 2:34 PM

As Ben mentioned, this is because your system is trying to run the RANCID provisioning integration components which attempt to connect to RANCID at startup.

You should be able to avoid this by removing the /opt/opennms/lib/opennms-rancid-provisioning-adapter-1.11.0-SNAPSHOT.jar file from the system. However, this error message is harmless.

jcat March 15, 2012 at 7:53 AM

Well - it's annoying for me - but the pre-built debian packages seem to start without error.

I wondered whether it was because I was compiling using OpenJDK, so I've built it again with Sun java 6, but same error.
I can build and start 1.8.16 in the same way without any issues at all.

I'm not sure if this is a red herring, but because it's some kind of HTTP connection refusal, I checked netstat on the local server to see if there were any connection failures. I do see connection attempt that seems to fail as it's starting:

tcp6 0 1 10.97.2.233:38476 10.97.2.233:1099 SYN_SENT 6788/java

Is there any extra info I can provide that may help?

Cheers,
Just

jcat March 14, 2012 at 7:02 AM

Thanks for the fast response

I simply do:

./compile.pl
./assemble.pl -Dopennms.home=/opennms/current

As a test - I'm going to install from pre-built the packages and compare - I'll report back.

Cheers,
Just

Configuration

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

PagerDuty

Created March 13, 2012 at 3:50 PM
Updated January 27, 2017 at 4:20 PM
Resolved March 20, 2012 at 2:34 PM