Uploaded image for project: 'OpenNMS'
  1. OpenNMS
  2. NMS-12104

Flow rest results for top N queries are not returned in the correct order

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Meridian-2019.1.0, 25.1.0
    • Component/s: None
    • Security Level: Default (Default Security Scheme)
    • Labels:
    • Epic Link:
    • Sprint:
      Horizon 2019 - September 25th

      Description

      When we do a Top N query (for hosts, conversations, or applications) the results we get back are not ordered correctly.

      We compose two queries, one to get the top N (which is returned in the correct order), followed by another query using that top N result and returning series data. The series data appears to be sorted by the default which is the # of flow documents associated.

      The # of flow documents appears to be highly correlated to the parameters dictating the top N in the first place so most of the time results will appear correctly ordered, but not always.

      Here is an example of results that are not in order:

       

      http://localhost:8980/opennms/rest/flows/hosts?N=10&end=1559939526079&includeOther=false&start=1559917926079
      
      {
          "start": 1559917926079,
          "end": 1559939526079,
          "headers": [
              "host",
              "Bytes In",
              "Bytes Out"
          ],
          "rows": [
              [
                  "172.20.50.109",
                  105381372,
                  0
              ],
              [
                  "172.217.13.164",
                  10761011,
                  0
              ],
              [
                  "172.20.1.22",
                  6993746,
                  0
              ],
              [
                  "23.41.248.2",
                  1237316,
                  0
              ],
              [
                  "104.88.157.41",
                  5639270,
                  0
              ],
              [
                  "3.89.174.207",
                  4792546,
                  0
              ],
              [
                  "192.30.253.112",
                  4847232,
                  0
              ],
              [
                  "172.217.13.142",
                  3963527,
                  0
              ],
              [
                  "172.217.13.193",
                  3690937,
                  0
              ],
              [
                  "69.147.64.33",
                  2783200,
                  0
              ]
          ]
      }
      

      Note the 4th value of 1237316 is smaller than the 5th value of 5639270. I think the results should be returned from the ReST API in the the same order as the top N are determined so that clients do not need to apply sorting.

      I suspect it should be fairly easy to order the results given that we retrieved the order from the first part of the composed query. We can reorder the results from the second part of the query based on the first.

       

      Here is the TopN hosts result from ES for the above ReST result showing sorting by flow document:

      {{ "took": 9, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 10141, "max_score": 0.0, "hits": [] }, "aggregations": { "grouped_by": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "172.20.50.109", "doc_count": 10141, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 10141, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 10141, "value": 1.053606188763654E8 }] } }] } }, { "key": "3.89.174.207", "doc_count": 458, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 458, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 458, "value": 4792546.066042456 }] } }] } }, { "key": "172.217.13.142", "doc_count": 230, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 230, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 230, "value": 3956770.1529989513 }] } }] } }, { "key": "172.20.1.22", "doc_count": 134, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 134, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 134, "value": 6993746.0 }] } }] } }, { "key": "172.217.13.164", "doc_count": 133, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 133, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 133, "value": 1.0761011E7 }] } }] } }, { "key": "192.30.253.112", "doc_count": 72, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 72, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 72, "value": 4847232.0 }] } }] } }, { "key": "172.217.13.193", "doc_count": 67, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 67, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 67, "value": 3690937.0 }] } }] } }, { "key": "23.41.248.2", "doc_count": 40, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 40, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 40, "value": 1237316.0076530504 }] } }] } }, { "key": "104.88.157.41", "doc_count": 8, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 8, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 8, "value": 5639270.0 }] } }] } }, { "key": "69.147.64.33", "doc_count": 8, "direction": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [{ "key": "ingress", "doc_count": 8, "bytes": { "buckets": [{ "key_as_string": "2019/06/07 14:32:06", "key": 1559917926079, "doc_count": 8, "value": 2783200.0 }] } }] } }] } }}

       

        Attachments

          Activity

            People

            • Assignee:
              fooker Dustin Frisch
              Reporter:
              mbrooks Matthew Brooks
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: