elastic/elasticsearch

Allocation filtering works only on delta settings - old excluded attributes are not honored

Open

#55,815 opened on Apr 27, 2020

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Java (25,882 forks)batch import
:Core/Infra/Settings>bugTeam:Core/Infrahelp wanted

Repository metrics

Stars
 (76,700 stars)
PR merge metrics
 (Avg merge 2d) (1,000 merged PRs in 30d)

Description

Elasticsearch version (bin/elasticsearch --version): 7.1, 7.4 and likely other versions

Description of the problem including expected versus actual behavior: Allocation filtering works only on delta settings - ie. old filter attributes of the same family(include/exclude/require) are not honored

This bug occurs because FilterAllocationDecider is recreating its include/exclude/require filters from the settings update consumer. The AffixMapUpdateConsumer is only passing setting deltas

So with 2 independent exclusions, we end up having a cluster where /_cluster/settings has multiple attributes for shard exclusion but the decider acts only on few of them which were part of the last exclude delta

This does not look like expected behavior. Till we have a fix, I would also like to get your thoughts on updating https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-filtering.html to indicate such cases

Steps to reproduce:

  1. Decommission a zone - relocate shards from all nodes in a particular zone
curl -H "Content-Type: application/json" -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude.zone" : "us-east-1a"
    }
}'

If this is the only excluded attribute, everything works as expected

  1. Decommission a rack - relocate shards from all nodes in a particular rack
curl -H "Content-Type: application/json" -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude.rack" : "rack-id"
    }
}'
  1. At this point, the cluster settings will have both
"cluster.routing.allocation.exclude.rack" : "rack-id"
"cluster.routing.allocation.exclude.zone" : "us-east-1a"

But because the delta on second update was only for rack, shards will start relocating into the excluded us-east-1a zone

Contributor guide