Slow LinkdTopologyProvider/EnhancedLinkdTopologyProvider in bigger enviroments

After an upgrade from 1.13.4 to 16.0.2 in an OpenNMS setup with 8000 nodes and 12000 links detected by EnhancedLinkd it took 15-30 minutes to open the TopologyMap. In 1.13.4 with Linkd, opening the Topology Map took only a few seconds.

I had a look in karaf.log and in EnhancedLinkdTopologyProvider and saw, that the full topology was initially loaded on OpenNMS startup with the log lines

loadtopology: parsing lldp link with id...
loadtopology: adding cdp link...
...

This took around 20minutes. If I open the Topology Map in the OpenNMS WebUI, I saw these log lines in karaf.log and the loading of the full topology again, which takes another 20 minutes to complete and to show up the map.

This also happens if I use the LinkdTopologyProvider in that environment in 15.0.2. I did some further investigation and saw a lot of single database queries. Some of them were really unnecessary like queries of the cdpelement/lldpelement/ospfelement tables when using LinkdTopologyProvider.

I created a (really bad ) patch for the LinkdTopologyProvider in 15.0.2 (see attached file) and replaced the Hibernate stuff with a single SQL query which gets all links from the datalinkinterface table and some information about the connected nodes. This speeded up the loading of the topology map in my environment from 20 minutes to a few seconds.