Fixed in release-16.0.4 with 87a587810dbd533c657faa29babeec79cf742ec7

The deadlocks haven't reappeared since demo was updated with this change (they normally would have by now). I'll go ahead and mark this as fixed, and we'll revisit it again if it reoccurs.

I think I've isolated this one.

We're seeing these deadlocks occur (updates to the assets table) in contexts where we should only be reading values.

I was able to reproduce this by calling org.opennms.features.vaadin.surveillanceviews.service.DefaultSurveillanceViewService#getNodeRtcsForCategories() from a integration test. Hibernate logs revealed that the 'userLastModified' field was being marked as dirty, and as a result, it would attempt to update the row in question.

The 'userLastModified' field is of type 'char(20)', but the default value in the OnmsAssetRecord object is an empty string. char types are always of a fixed size, and padded with spaces in PostgreSQL.

Changing the type of the 'userLastModified' field to 'varchar(20)' while keeping the default value of an empty string resolved the issue with the "update on read". I believe that this should fix the related deadlocks as well.

I have made a series of thread dumps on the server with the GUI and the “core”- Server with all other daemons. The dumps are made in an interval from ca. 30 seconds (the dump need some time).
There are some threads in block state for over 4 minutes (I add timestamps in the first line of every dump) but I cant say if that is normal or not.

Please get a thread dump from the system whenever it is slow and we can look at that a see what is going on inside the system to diagnose the slowness.

I don't think that database deadlocks would contribute much to slowing the system down. In your logs, the deadlock condition lasts for a couple of seconds and then goes away. I think that the worst effect of the deadlocks would be that Provisiond may not be updating some data properly or adding nodes to the database when the deadlocks occur.