yugabyte/yugabyte-db

[DocDB] Retrigger delete tablet tasks after recovering from a network partition

Open

#22,212 opened on Apr 30, 2024

View on GitHub
 (1 comment) (0 reactions) (1 assignee)C (1,003 forks)batch import
area/docdbgood first issuekind/bugpriority/medium

Repository metrics

Stars
 (8,229 stars)
PR merge metrics
 (Avg merge 17d 21h) (92 merged PRs in 30d)

Description

Jira Link: DB-11132

Description

If a tserver is disconnected from other nodes (but not shut down), its tablet peers will be kicked out of their RAFT groups and the load balancer will add new peers. When it reconnects to the other nodes, the master / tservers will not send a delete for the tablets that were kicked out (so they are stuck in the RUNNING state).

The tablet peers should be deleted. Possible fixes:

  1. Force a full heartbeat after a tserver is marked dead
  2. Periodically do full heartbeats from all tservers
  3. If a tserver heartbeats to a member of the active quorum with a newer config, tell the stale tserver to do a full heartbeat (which will cause the master to send DeleteTablet requests)

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.

Contributor guide

Research direction
Tech stack
Domain
Issue type
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Prerequisites
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.