Requisition list very slow to display
Description
Environment
Acceptance / Success Criteria
Lucidchart Diagrams
Activity
Alejandro Galue January 5, 2015 at 4:25 PM
commit 84eb213e53b0946476ee0f922a3f325d950ce7ab
Author: Alejandro Galue <agalue@opennms.org>
Date: Tue Dec 23 10:27:09 2014 -0500
New Feature: Managing Foreign-Source Repository Strategy
Adding required insfrastructure to select requisition strategy.
A factory class has been created in order to dynamically select the
strategy used to manage requisition based on the value of a system
property called org.opennms.provisiond.repositoryImplementation,
declared on opennms.properties. The default value is file (the currently
used strategy).
Available strategies for Production:
file (default)
fastFile (recommended for fastest response, at expense of memory resources)
Available experimental strategies:
fused
fastFused
queueing
fastQueueing
caching
fastCaching
The "fast" version means use the implementations for "fastFile" in
conjunction with ether fused, queueing or caching.
The default qualifier (i.e. "deployed", "pending") for the
ForeignSourceRepository interface are going to use the factory
to obtain the configured strategy.
The DirectoryWatcher class (used by the "fastFile" strategy) is going to
create and maintain a tread for the directory to watch using Java7's
directory watcher feature. The reason for the thread is because and
infinite loop is required to wait for file changes.
Tests:
Tested on a live OpenNMS with more than 500 requisitions and more than
10000 nodes in total. The response time of the Requisitions ReST API is
almost instantaneously. Of course a cache of the requisitions is kept on
memory in order to accelerate the response time.
Alejandro Galue December 10, 2014 at 4:17 PM
Here is a suggestion:
Considering that every related requisition operation can be done through the ReST API, and considering that we can educate users to avoid messing up with the content of /opt/opennms/etc/imports, we can just persist the requisition on the DB, and retrieve the data from it (and stop using that directory and maintain the XMLs).
The current WebUI (ignoring for a moment that it is completely useless for big requisitions and/or lots of requisitions) is not aware of the XML on the imports directory either, so just removing the need of unmarshalling the XML could improve its response time (I think).
We are still going to have the delay of marshalling the objects to XML to generate the ReST response, but I think that could be an improvement.
Alejandro Galue November 18, 2014 at 3:58 PM
Considering that one of the slowest component is the DOM Parsing in Java (as I demonstrated comparing the standard XML Collector with and without VTD-XML), it looks like reading the XML from the pending directory, parse it through DOM using JAXB and then generate the exact same XML again to return it through ReST seems to be very expensive.
Alejandro Galue November 18, 2014 at 3:39 PM
Based on Jeff's test, I've created a VM with Vagrant (with CentOS 7, RRDtool, Oracle JDK 7u71, PostgreSQL 9.3.5, NodeJS, OpenNMS 14.0.1 and my AngularJS Application for handling Requisitions). Then I've executed the following:
Using FireBug, the time spent loading the requisitions and rendering the UI using my Angular App and the ReST API for requisitions was less than 11 seconds (with 50 requisitions). The overhead of processing the JSON data returned from the OpenNMS server via ReST was just 37ms.
With 100 requisitions the total time was about 21 seconds (expending less than 50 ms processing the JSON data).
With 200 requisitions the total time was about 42 seconds (expending less than 63 ms processing the JSON data).
Do you see a pattern here ?
Conclusion: the most expensive operation is retrieving the pending and deployed requisitions through ReST. The time spent processing the JSON data depends on the complexity and the size of the requisitions. On this experiment all the requisitions have only one node and none of them have been deployed (they are pending).
The thing is, the Angular app is still a lot faster than the current UI
Alejandro Galue November 7, 2014 at 4:52 PM
Here is the solution:
https://github.com/agalue/OpenNMS-UI-Requisitions
It only works with OpenNMS 14 (or newer).
The "slowest" part is request big requisitions through the ReST API due to (I think) parsing the XML to an object, and then parsing it again to XML to return the data through HTTP. This affects not only my AngularJS app, but also the speed of provision.pl (because it uses the ReST API), and the regular WebUI (because it uses the underlying classes to access the requisitions, similar to the ReST API).
In my opinion, the only way to accelerate the process is avoid involving XML to read from and store XML. If we say the ReST API is "the way" to manipulate requisitions, we can store it on a different format.
To create fifty requisitions and get them all flushed to the filesystem, start OpenNMS and then do:
for loop
Now ready a stopwatch and visit Admin -> Manage Provisioning Requisitions. Or time it using curl:
curl
On my system with 50 single-node requisitions, the time is about 32 seconds. With 100 single-node requisitions it's about 71 seconds, so something like a linear increase proportional to the requisition count.
This problem appears unrelated to my commit of 02 Oct which gives a minor facelift to the list; backing out those changes and doing a clean build / clean install makes no difference to the delay.
While the page is loading, I see N+4 of the following messages where N is the number of requisitions:
The pattern is interesting, I see two of them very close together, followed by a pause of about 2N seconds, then the remaining N+2 arrive at a steady rate of about 5 per second. These stats hold up for the two cases I've tested, N=50 and N=100, so don't put a ton of stock in them.