kubernetes/minikube

Ambassador Addon fails to start

Open

#10,093 opened on Jan 5, 2021

View on GitHub
 (5 comments) (4 reactions) (0 assignees)Go (5,222 forks)batch import
area/addonshelp wantedkind/buglifecycle/frozenpriority/backlog

Repository metrics

Stars
 (31,799 stars)
PR merge metrics
 (Avg merge 12d 19h) (43 merged PRs in 30d)

Description

Steps to reproduce the issue:

  1. (minikube delete)
  2. minikube start
  3. minikube addons enable ambassador
  4. Wait :P
  5. minikube kubectl -- get pods --namespace ambassador
  6. (minikube kubectl -- logs ambassador-7484d7d786-f2rf5 --namespace ambassador)

Enabling the Ambassador-Addon does work, however the 3 Pods (not the ambassador-operator) won´t start properly. Instead they log the following message over and over:

Pod-Log: E0105 16:48:22.689662 1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

It seems as if the problem is caused by the new Ambassador-Version (1.10.0). In Version 1.9.1 everything works fine.

Full Pod-Log:

2021/01/05 16:47:48 Started Ambassador
2021/01/05 16:47:49 AMBASSADOR_CLUSTER_ID=40e97ad7-c84a-5e4d-aad1-e4ad25ca2034
time="2021-01-05 16:47:49" level=info msg="[pid:21] started command []string{\"diagd\", \"/ambassador/snapshots\", \"/ambassador/bootstrap-ads.json\", \"/ambassador/envoy/envoy.json\", \"--notices\", \"/ambassador/notices.json\", \"--port\", \"8004\", \"--kick\", \"kill -HUP 1\"}" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:143" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stdin  < not logging input read from file /dev/stdin" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:145" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stdout > not logging output written to file /dev/stdout" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:148" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stderr > not logging output written to file /dev/stderr" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:151" CMD=entrypoint PID=1 THREAD=/diagd
E0105 16:47:49.373264       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE disabled, not initializing cache
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: WILL NOT update Mapping status
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: thread count 25, listening on 0.0.0.0:8004
E0105 16:47:50.969162       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-05 16:47:51 diagd 1.10.0 [P21TMainThread] INFO: Ambassador 1.10.0 booted
[2021-01-05 16:47:51 +0000] [21] [INFO] Starting gunicorn 20.0.4
[2021-01-05 16:47:51 +0000] [21] [INFO] Listening at: http://0.0.0.0:8004 (21)
[2021-01-05 16:47:51 +0000] [21] [INFO] Using worker: threads
[2021-01-05 16:47:51 +0000] [27] [INFO] Booting worker with pid: 27
2021-01-05 16:47:51 diagd 1.10.0 [P27TAEW] INFO: starting Scout checker and timer logger
2021-01-05 16:47:51 diagd 1.10.0 [P27TAEW] INFO: starting event watcher
E0105 16:47:53.660978       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:47:57.563608       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021/01/05 16:47:59 Memory Usage 0.07Gi (12%)
    PID 1, 0.06Gi: busyambassador entrypoint 
    PID 21, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
    PID 27, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
E0105 16:48:05.266506       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:48:22.689662       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:49:02.902865       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:49:58.781330       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

Running Pods:

NAME                                  READY   STATUS    RESTARTS   AGE
ambassador-7484d7d786-f2rf5           0/1     Running   0          9m20s
ambassador-7484d7d786-f6648           0/1     Running   0          9m20s
ambassador-7484d7d786-mpg4h           0/1     Running   0          9m20s
ambassador-operator-fcb94f945-kckgx   1/1     Running   0          9m45s

Addons which are also enabled:

  • storage-provisioner
  • default-storageclass

Specs: Driver: Parallels CPUs: 12 RAM: 12GB Disk: 30GB OS: macOS 11.1

Confirmend the error also exists with the docker driver.

Contributor guide

Research direction
Investigate the Ambassador addon's RBAC configuration in minikube and update the ClusterRole or Role to include permissions for listing ingressclasses in networking.k8s.io.
Tech stack
go
Domain
backendinfrastructure
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Kubernetesminikube
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
80