GoogleContainerTools/skaffold

Mysterious Warn logs in dev loop when container is terminated and watching for changes

Open

#5,565 opened on Mar 17, 2021

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (1,416 forks)batch import
area/status-checkbugbash/q4-2022help wantedkind/frictionpriority/p2

Repository metrics

Stars
 (12,822 stars)
PR merge metrics
 (Avg merge 3d 6h) (16 merged PRs in 30d)

Description

I am testing a scenario where users have a deployed container fails before status check phase is complete.

As expected Skaffold

  1. reports the container is terminated with exit code.
  2. Tries fetching logs which arent there.
  3. quits the status check loop and watches for changes

Unexpected:

  1. I see the following warn messages in the watching phase WARN[0046] exit status 1
  2. WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed is incorrect because the deploy already happened and we don't clean up.
Waiting for deployments to stabilize...
 - deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
WARN[0045] exit status 1                                
 - deployment/leeroy-app: container leeroy-app terminated with exit code 2
    - pod/leeroy-app-794c7c586d-2qrh2: container leeroy-app terminated with exit code 2
      > Error retrieving logs for pod leeroy-app-794c7c586d-2qrh2. Try `kubectl logs leeroy-app-794c7c586d-2qrh2 -n default -c leeroy-app`
 - deployment/leeroy-app failed. Error: container leeroy-app terminated with exit code 2.
WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed 
Watching for changes...
WARN[0046] exit status 1                                
WARN[0060] exit status 1                                
WARN[0090] exit status 1  

How to reproduce

  1. Add a lag in status check so the container has time to crash before status check ends.
+++ b/pkg/skaffold/deploy/status/status_check.go
@@ -97,6 +97,7 @@ func NewStatusChecker(cfg Config, labeller *label.DefaultLabeller) Checker {
 // Run runs the status checks on deployments and pods deployed in current skaffold dev iteration.
 func (s statusChecker) Check(ctx context.Context, out io.Writer) error {
        event.StatusCheckEventStarted()
+       time.Sleep(1*time.Second)
        errCode, err := s.statusCheck(ctx, out)
        event.StatusCheckEventEnded(errCode, err)
        return err
  1. LOCAL=true make

  2. run skaffold dev

  3. After the first deploy, add a NPE to mimic container crash

--- a/examples/microservices/leeroy-app/app.go
+++ b/examples/microservices/leeroy-app/app.go
@@ -12,6 +12,7 @@ func handler(w http.ResponseWriter, r *http.Request) {
 
 func main() {
        log.Print("leeroy app server ready")
+       fmt.Fprintf(nil, "leeroooooy app!!\n")
        http.HandleFunc("/", handler)
        http.ListenAndServe(":50051", nil)
 }

Contributor guide