kubernetes/kubernetes

kubelet parameter(eviction-max-pod-grace-period ), not work as expected like officical comment.

Open

#118,172 opened on May 22, 2023

View on GitHub
 (37 comments) (0 reactions) (8 assignees)Go (43,066 forks)batch import
good first issuehelp wantedkind/bugkind/documentationpriority/backlogpriority/important-longtermsig/nodetriage/accepted

Repository metrics

Stars
 (122,268 stars)
PR merge metrics
 (Avg merge 28d 17h) (343 merged PRs in 30d)

Description

What happened?

Hey brothers , feedback a confusing point about kubelet soft eviction:

Just like kubelet --help show: --eviction-max-pod-grace-period int32 Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value.

I guess if this parameter is set to negative ,such as -1, soft eviction would use pod specified value(TerminationGracePeriodSeconds),

But When I try to evict pod by creating node pressure(such as memory pressure), I found it's always -1 send to CRI runtime. Then pod container stopped immediately with sigkill, exit 137.

In short, this parameter: (1)set as active number:work as expected; (2)no set, kubelet read its default number:0, work as expected; (3)set as negative number: not work as kubelet help show.

Thanks for your response. May it be a bug ?

What did you expect to happen?

--eviction-max-pod-grace-period set to negative, then kubelet syncPod logic send pod‘s TerminationGracePeriodSeconds as gracePeriod to CRI runtime to stop container.

How can we reproduce it (as minimally and precisely as possible)?

(1) set this parameter to a negative number, such as -1; (2) config kubelet soft eviction ; (3) running a pod asking a lot memory , to trigger node memory pressure; (4) check CRI runtime log, such as containerd ; (5) then you will find timeout parameter -1 sent to pod container , then container exit with code 137(sigkill);

Anything else we need to know?

I running these tests on Tencent Cloud TKE.

And I thought it seem not related to Cloud Service Provider.

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.3-tke.34", GitCommit:"61cc96d2f7e9277e89e29b4b04d045f27d6e75df", GitTreeState:"clean", BuildDate:"2023-03-08T07:40:04Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.3-tke.32", GitCommit:"8ca5807ae095365d68754a2ba20b10bbe5a8998c", GitTreeState:"clean", BuildDate:"2022-10-08T04:00:33Z", GoVersion:"go1.12.14", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

Install tools

Container runtime (CRI) and version (if applicable)

containerd --version containerd github.com/containerd/containerd v1.4.3-tke.2 a11500cbfa0b4d2fc9b905e03c35f349ef5b1a9f

Related plugins (CNI, CSI, ...) and versions (if applicable)

Contributor guide