Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-13507

Enlinkd API response extremely slow for some nodes

    XMLWordPrintable

Details

    • 13
    • Horizon 2021 - Dec 1-15
    • Backlog
    • 698

    Description

      For some nodes with complicated topologies and/or lots of links, responses from the enlinkd rest api are very slow.  As best I can tell, the bottleneck is jetty's processing of the response, not the database executing the associated queries.

       

      Stack traces for the jetty qtp threads processing the requests are similar to:

      stackTrace:
      java.lang.Thread.State: RUNNABLE
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
      at org.hibernate.property.BasicPropertyAccessor$BasicGetter.get(BasicPropertyAccessor.java:172)
      at org.hibernate.tuple.entity.AbstractEntityTuplizer.getIdentifier(AbstractEntityTuplizer.java:227)
      at org.hibernate.persister.entity.AbstractEntityPersister.getIdentifier(AbstractEntityPersister.java:3876)
      at org.hibernate.event.def.DefaultFlushEntityEventListener.checkId(DefaultFlushEntityEventListener.java:80)
      at org.hibernate.event.def.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:190)
      at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:147)
      at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:219)
      at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:99)
      at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:58)
      at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1185)
      at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1261)
      at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
      at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890)
      at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate$3.doInHibernate(AbstractDaoHibernate.java:199)
      at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:410)
      at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:341)
      at org.opennms.netmgt.dao.hibernate.AbstractDaoHibernate.findUnique(AbstractDaoHibernate.java:204)
      at org.opennms.netmgt.dao.hibernate.SnmpInterfaceDaoHibernate.findByNodeIdAndIfIndex(SnmpInterfaceDaoHibernate.java:50)
      at org.opennms.web.enlinkd.EnLinkdElementFactory.create(EnLinkdElementFactory.java:352)
      at org.opennms.web.enlinkd.EnLinkdElementFactory.convertFromModel(EnLinkdElementFactory.java:373)
      at org.opennms.web.enlinkd.EnLinkdElementFactory.getCdpLinks(EnLinkdElementFactory.java:339)
      at jdk.internal.reflect.GeneratedMethodAccessor843.invoke(Unknown Source)
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
      at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
      at com.sun.proxy.$Proxy210.getCdpLinks(Unknown Source)
      at org.opennms.web.rest.v2.NodeLinkRestService.getCdpLinks(NodeLinkRestService.java:123)
      at org.opennms.web.rest.v2.NodeLinkRestService.getCdpLinks(NodeLinkRestService.java:119)
      at jdk.internal.reflect.GeneratedMethodAccessor842.invoke(Unknown Source)
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
      at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
      at com.sun.proxy.$Proxy264.getCdpLinks(Unknown Source)
      at jdk.internal.reflect.GeneratedMethodAccessor842.invoke(Unknown Source)
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
      at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
      at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
      at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
      at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
      - locked <0x00000000e28abfd8> (a org.apache.cxf.phase.PhaseInterceptorChain)
      at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
      at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
      at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
      at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
      at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
      at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
      at org.opennms.container.web.bridge.proxy.ProxyFilter.doFilter(ProxyFilter.java:110)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.extremecomponents.table.filter.AbstractExportFilter.doFilter(AbstractExportFilter.java:49)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.opennms.web.servlet.StoreRequestPropertiesFilter.doFilter(StoreRequestPropertiesFilter.java:79)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
      at org.opennms.web.springframework.security.AuthFilterEnabler.doFilter(AuthFilterEnabler.java:115)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
      at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
      at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:231)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.opennms.web.security.JSessionIdNoCacheFilter.doFilter(JSessionIdNoCacheFilter.java:68)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
      at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      at org.opennms.netmgt.jetty.MDCHandler.handle(MDCHandler.java:46)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      at org.eclipse.jetty.server.Server.handle(Server.java:516)
      at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
      at org.eclipse.jetty.server.HttpChannel$$Lambda$1083/0x0000000841d44440.dispatch(Unknown Source)
      at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
      at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
      at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)

       

      In a test environment with an older copy of the customer database, I have identified nodes with high response times:

       

      Nodeid Seconds
      811 11.21
      9320 11.103
      8987 11.01
      4936 10.511
      10989 10.122
      11324 10.077
      7078 9.889
      8451 9.522
      3867 9.293
      7788 9.188
      5905 9.019

       

      Attachments

        Issue Links

          Activity

            People

              swachter Stefan Wachter
              dino2gnt Dino Yancey
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.