Repository metrics
- Stars
- (76,700 stars)
- PR merge metrics
- (Avg merge 2d) (1,000 merged PRs in 30d)
Description
If bucket_sort orders buckets based on a sub-agg that is "empty" (because it received no values and returns NaN/null/etc), that bucket is removed from the final list. This happens because the default gap_policy for pipeline aggs is skip, and the agg will not include buckets if the value is NaN && skip
Changing the gap policy to insert_zeros "fixes" the issue, but I'm thinking it doesn't make sense to support gap_policy in bucket_sort at all. The gap policy is for filling gaps so that sequential operations (derivative, etc) can continue to work. BucketSort is just sorting existing buckets and potentially truncates. Skipping empty buckets is wrong, and "inserting zeros" doesn't make sense.
I think we should just drop support for that parameter from BucketSort.
In this example we can see that ordering by the percentiles sub-agg omits one of the buckets, because one of the buckets is empty (no docs) and thus the percentiles agg is NaN internally. Combined with the default skip it means the bucket is removed:
PUT comments
{"mappings":{"comment":{"properties":{"location_i":{"type":"integer"},"type_i":{"type":"integer"},"duration":{"type":"integer"}}}}}
PUT comments/comment/1
{"location_i":1,"type_i":1,"duration":100}
PUT comments/comment/2
{"location_i":2,"type_i":2,"duration":50}
POST comments/_search
{
"size": 0,
"aggs": {
"locations": {
"terms": {
"field": "location_i",
"size": 10
},
"aggs": {
"onlyTypeOne": {
"filter": {
"term": {
"type_i": 1
}
},
"aggs": {
"medianDuration": {
"percentiles": {
"field": "duration",
"percents": [
50
]
}
}
}
},
"location_sort": {
"bucket_sort": {
"sort": [
{
"onlyTypeOne>medianDuration.50": {"order": "asc"}
}
],
"size": 10
}
}
}
}
}
}
returning:
{
"aggregations" : {
"locations" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 1,
"doc_count" : 1,
"onlyTypeOne" : {
"doc_count" : 1,
"medianDuration" : {
"values" : {
"50.0" : 100.0
}
}
}
}
]
}
}
}
But if change to insert_zeros we get back the values expected:
POST comments/_search
{
"size": 0,
"aggs": {
"locations": {
"terms": {
"field": "location_i",
"size": 10
},
"aggs": {
"onlyTypeOne": {
"filter": {
"term": {
"type_i": 1
}
},
"aggs": {
"medianDuration": {
"percentiles": {
"field": "duration",
"percents": [
50
]
}
}
}
},
"location_sort": {
"bucket_sort": {
"sort": [
{
"onlyTypeOne>medianDuration.50": {"order": "asc"}
}
],
"size": 10,
"gap_policy": "insert_zeros"
}
}
}
}
}
}
returning:
{
"aggregations" : {
"locations" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 2,
"doc_count" : 1,
"onlyTypeOne" : {
"doc_count" : 0,
"medianDuration" : {
"values" : {
"50.0" : null
}
}
}
},
{
"key" : 1,
"doc_count" : 1,
"onlyTypeOne" : {
"doc_count" : 1,
"medianDuration" : {
"values" : {
"50.0" : 100.0
}
}
}
}
]
}
}
}
Found from #36402