Looks like there is two other ways to do this, using a script at query time or using a copy_to field at index time.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_multi_field_terms_aggregation

Using copy_to doesn't solve the issue about knowing the format of a convo key at index time.

Using script looks like it will be much less performant and since we have many documents likely will not be adequate.

So based on what I've found it seems that there isn't a better solution compared to how we are currently doing it.

This post describes a use case similar to ours where the recommended solution is to use terms aggregation with partitioning because a composite aggregation cannot do what is needed.

https://discuss.elastic.co/t/composite-aggregation-sorting/156936

Here is the query I was trying. It selects the buckets as we want but it only performs the sorting on the buckets that it selected. So If I query for 5, I get 5 results which are ordered based on natural order of the field values, then those 5 results have the total bytes agg applied and are sorted by that.

I gave this a try and it appears that this won't work for our use case. The issue is that we we cannot use the total bytes sum aggregation the same way. We can only apply the sum to the total bytes of the N buckets that were already selected and the only way we can order the buckets we selected is by the natural order of fields we include in grouping the buckets by. Doing that is not useful.

So as far as I can tell there is no way, using composite aggregation, to ask for the top N "buckets" based on the aggregation yielding the sum of the total bytes. We can only sort on a value that is contained in the bucket.