Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Limit CollectionSet size in Kafka Producer to less than 1MB.

Description

Currently, Kafka Producer for metrics doesn't split message and put all the metrics it receives for a resource into one message.

Sometimes this may be over 1MB which may result in MessageSizeTooLargeException exceptions in Kafka.

To resolve this, we could split resources into several chunks and send with the same timestamp.

Acceptance / Success Criteria

None

Lucidchart Diagrams

Activity

Show:

Chandra Gorantla July 6, 2021 at 6:18 PM

Re-targeted this to develop.  Will backport if needed

Chandra Gorantla June 29, 2021 at 6:14 PM
Edited

I don't think there wouldn't be any issue splitting CollectionSet..

Above PR splits CollectionSet with too many resources into multiple CollectionSet with same timestamp. 

I don't see any impact on customers even if they have custom code.

message CollectionSet { int64 timestamp = 1; repeated CollectionSetResource resource = 2; }

Alejandro Galue June 29, 2021 at 5:22 PM

Considering that all the customers that are currently using the Kafka Producer for metrics reconfigured their brokers (and the producer itself) to allow large messages to avoid this issue (as this is a known problem since day one), and all those customers already wrote code with this in mind, is the split operation going to be enabled by default? If so, what kind of changes would be required for the customers to use it?

I hope it is not enabled by default (even if it is a useful enhancement), as you're targeting an existing Meridian version.

Chandra Gorantla June 29, 2021 at 4:26 PM

Fixed

Details

Assignee

Reporter

Sprint

Fix versions

Priority

PagerDuty

Created June 23, 2021 at 9:46 PM
Updated March 8, 2024 at 6:58 PM
Resolved July 6, 2021 at 9:11 PM

Flag notifications