I checked the dropped samples counter, it was 0.

I also didn't see any significant load, I think we are good. But I think we should test again once we have a concrete scenario

Test scenario:

• check out release-26.x (base line) / NMS-12731-perf-3 (performance improvements)

• build

• run empty database: sudo docker run -p 5432:5432 -e POSTGRES_PASSWORD= postgres:11.4

• enable timeseries integration layer in opennms.properties

• start system

• install inmemory plugin via karaf

• connect to JMX console

• run commans ` stress-metrics --interfaces 5 --strings 2 --interval 30 --nodes 10000 --threads 10 --groups 5 --attributes 5

• watch the counters

I moved the different bottlenecks to subtasks and made this ticket the umbrella for all optimizations.

Talking to @Jesse White ,

the goal is to make sure the integration layer is as performant as possible. The current structure is optimized for Newts / Cassandra, not for a tag based storage. We want to be able to wrote 40k samples / second.

In order to stress test and profile the system we can use the following commands:
One off:
admin@opennms> collect --persist --node 2 org.opennms.netmgt.collectd.SnmpCollector 127.0.0.1
Stress:
admin@opennms> stress-metrics --interfaces 5 --strings 2 --interval 30 --nodes 10000 --threads 10 --groups 5 --attributes 5

I looked into the different caches we have in the TimeseriesIntegrationLayer:

• SearchableResourceMetadataCache: used byTimeseriesResourceStorageDao. Stores

• TimeseriesSearcher.metricsUnderResource: (new) caches all Metrics that can be found under a resource (by a wildcard search). Caches results coming from TimeseriesStorage implementation.

• TimeSeriesMetaDataDao.cache: caches all attributes associated with a resourceId. Caches results from the database.