Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Sink API: Persistent Off-Heap Storage

Description

We currently support off-heap storage for Sink messages on Minion when it can't reach the broker (added in https://issues.opennms.org/browse/NMS-10942).

The current solution uses H2 database as key value store (MVStore) for an in-memory (off-heap) solution. Some of the drawback of this solution are:

  • Large memory overhead per message

  • Messages are lost on when the Minion is restarted

We now have additional requirements to make the messages persist across Minion restarts, while continuing to make a best effort to preserve the order.

Here we explore options to replace the current H2 implementation with one that is backed by RocksDB.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Matthew Brooks January 24, 2020 at 1:54 PM

Ended up merging to 25.1.3.

Jesse White January 21, 2020 at 9:38 PM

Correct - the change affects must of the Sink API code, so we'll target 26.0.0.

Alejandro Galue January 21, 2020 at 9:18 PM

The PR targets develop but the Fix Version is set to 25.1.3. It looks like it should be 26.0.0, is this correct?

Matthew Brooks January 14, 2020 at 7:15 PM

Opened the following issues as a result of testing this feature:

https://issues.opennms.org/browse/NMS-12477

https://issues.opennms.org/browse/NMS-12478

Matthew Brooks January 14, 2020 at 7:14 PM

In review now is an alternative implementation that replaces the H2 db impl as well as the Rox db impl that was under development. There is now one queue implementation that writes off-heap entries to the disk.

Fixed

Details

Assignee

Reporter

Components

Sprint

Fix versions

Priority

PagerDuty

Created February 27, 2019 at 4:44 AM
Updated January 24, 2020 at 1:54 PM
Resolved January 24, 2020 at 1:53 PM