PR: https://github.com/OpenNMS/opennms/pull/3025

I moved the different bottlenecks to subtasks and made this ticket the umbrella for all optimizations.

Talking to @Jesse White ,

the goal is to make sure the integration layer is as performant as possible. The current structure is optimized for Newts / Cassandra, not for a tag based storage. We want to be able to wrote 40k samples / second.

In order to stress test and profile the system we can use the following commands:
One off:
admin@opennms> collect --persist --node 2 org.opennms.netmgt.collectd.SnmpCollector 127.0.0.1
Stress:
admin@opennms> stress-metrics --interfaces 5 --strings 2 --interval 30 --nodes 10000 --threads 10 --groups 5 --attributes 5

I looked into the different caches we have in the TimeseriesIntegrationLayer:

• SearchableResourceMetadataCache: used byTimeseriesResourceStorageDao. Stores

• TimeseriesSearcher.metricsUnderResource: (new) caches all Metrics that can be found under a resource (by a wildcard search). Caches results coming from TimeseriesStorage implementation.

• TimeSeriesMetaDataDao.cache: caches all attributes associated with a resourceId. Caches results from the database.

TimeseriesMetaDataDao.storeMetadata cached only on reads, not on writes. I fixed that. We should see now a lot less writes I suppose...