[wip] [test-only] Test for controller ha #15321

skonto · 2024-06-10T16:08:07Z

Proposed Changes

See TestControllerHA is flakey #15238

Release Note

knative-prow · 2024-06-10T16:08:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: skonto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [skonto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2024-06-10T16:14:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.80%. Comparing base (62ce45c) to head (fbf67dd).
Report is 35 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15321      +/-   ##
==========================================
+ Coverage   84.76%   84.80%   +0.03%     
==========================================
  Files         218      218              
  Lines       13504    13504              
==========================================
+ Hits        11447    11452       +5     
+ Misses       1690     1686       -4     
+ Partials      367      366       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

skonto · 2024-06-10T18:36:45Z

/test istio-latest-no-mesh

skonto · 2024-06-11T07:48:03Z

The lease seems not being updated after pod restart, although it has been released. logs here and here downloaded-logs-20240611-000442.json I will try to print the leases after the restart to verify it.

"insertId": "am8bz9y3f27j2j3s",
"jsonPayload": {
  "caller": "leaderelection/context.go:167",
  "timestamp": "2024-06-10T19:27:08.396Z",
  "logger": "controller",
  "commit": "bbc55b6",
  "message": "\"controller-6cdfc667d6-4b8qq_2b1432a3-4afd-4c2a-b3ef-05bb4523f4ab\" has stopped leading \"controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.08-of-10\"",
  "knative.dev/pod": "controller-6cdfc667d6-4b8qq"

skonto · 2024-06-11T12:33:13Z

/test istio-latest-no-mesh

skonto · 2024-06-11T19:37:04Z

/test istio-latest-no-mesh

skonto · 2024-06-11T21:42:46Z

/test istio-latest-no-mesh

skonto · 2024-06-11T23:49:51Z

/test istio-latest-no-mesh

skonto · 2024-06-12T07:55:16Z

/test istio-latest-no-mesh

skonto · 2024-06-12T21:02:01Z

/test istio-latest-no-mesh

skonto · 2024-06-12T21:04:00Z

test/e2e-tests.sh

@@ -113,7 +113,8 @@ do
  if (( ! HTTPS )); then
    restart_pod_n "${SYSTEM_NAMESPACE}" "app=controller"
  fi
-  kubectl get leases -n "${SYSTEM_NAMESPACE}" -ojson | grep holder
+  sleep 120


After adding this delay here I cant reproduce it although does not seem related.

skonto · 2024-06-14T07:16:52Z

/test istio-latest-no-mesh

skonto · 2024-06-14T10:02:09Z

Reproduced it again but missed lease printing due to wrong ns, re-running. 🤞

skonto · 2024-06-14T15:03:18Z

/test istio-latest-no-mesh

skonto · 2024-06-14T19:27:08Z

/test istio-latest-no-mesh

skonto · 2024-06-14T21:26:22Z

/test istio-latest-no-mesh

skonto · 2024-06-15T07:32:56Z

/test istio-latest-no-mesh

skonto · 2024-06-15T09:40:16Z

/test istio-latest-no-mesh

skonto · 2024-06-15T17:42:49Z

Obviously not all leases are updated after pod restarting:

The second time the tests are executed all certificates leases but one are updated.

Looking at the downloaded-logs-20240615-204712.json it seems that the lock is never released

  {
    "insertId": "m77ofqucl0dxhuuf",
    "jsonPayload": {
      "caller": "leaderelection/context.go:167",
      "message": "\"controller-859dc45cfb-8l5m7_23d0e3f1-2b3d-4a5c-a870-948746801c09\" has stopped leading \"controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.00-of-10\"",
      "knative.dev/pod": "controller-859dc45cfb-8l5m7",
      "logger": "controller",
      "commit": "d071b17",
      "timestamp": "2024-06-15T10:21:27.869Z"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "kt2-4bc2f2da-af36-40c2-bad4-6cc8742ae-1",
        "location": "us-east1",
        "container_name": "controller",
        "namespace_name": "1357db35-8ecc-4cb7-b5e5-a7c9b7edbf4b",
        "pod_name": "controller-859dc45cfb-8l5m7",
        "project_id": "knative-boskos-76"
      }
    },
    "timestamp": "2024-06-15T10:21:27.869226666Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/kapp_k14s_io/app": "1718445170413216642",
      "k8s-pod/app": "controller",
      "k8s-pod/app_kubernetes_io/version": "devel",
      "compute.googleapis.com/resource_name": "gke-kt2-4bc2f2da-af36-40-default-pool-5aec12e9-v7f4",
      "k8s-pod/kapp_k14s_io/association": "v1.ae5f7406090d99cc0cb95abd8ded8439",
      "k8s-pod/app_kubernetes_io/component": "controller",
      "k8s-pod/pod-template-hash": "859dc45cfb",
      "k8s-pod/app_kubernetes_io/name": "knative-serving"
    },
    "logName": "projects/knative-boskos-76/logs/stdout",
    "receiveTimestamp": "2024-06-15T10:21:31.109539752Z"
  },
  {
    "insertId": "0yg652b5n1vnnjye",
    "jsonPayload": {
      "message": "Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io \"controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.00-of-10\": the object has been modified; please apply your changes to the latest version and try again",
      "pid": "1"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "project_id": "knative-boskos-76",
        "pod_name": "controller-859dc45cfb-8l5m7",
        "location": "us-east1",
        "cluster_name": "kt2-4bc2f2da-af36-40c2-bad4-6cc8742ae-1",
        "namespace_name": "1357db35-8ecc-4cb7-b5e5-a7c9b7edbf4b"
      }
    },
    "timestamp": "2024-06-15T10:21:27.869230187Z",
    "severity": "ERROR",
    "labels": {
      "k8s-pod/kapp_k14s_io/association": "v1.ae5f7406090d99cc0cb95abd8ded8439",
      "compute.googleapis.com/resource_name": "gke-kt2-4bc2f2da-af36-40-default-pool-5aec12e9-v7f4",
      "k8s-pod/app_kubernetes_io/component": "controller",
      "k8s-pod/app": "controller",
      "k8s-pod/app_kubernetes_io/name": "knative-serving",
      "k8s-pod/app_kubernetes_io/version": "devel",
      "k8s-pod/kapp_k14s_io/app": "1718445170413216642",
      "k8s-pod/pod-template-hash": "859dc45cfb"
    },
    "logName": "projects/knative-boskos-76/logs/stderr",
    "sourceLocation": {
      "file": "leaderelection.go",
      "line": "308"
    },
    "receiveTimestamp": "2024-06-15T10:21:31.157478025Z"
  }
]

cc @dprotaso

skonto · 2024-06-18T07:29:02Z

/test istio-latest-no-mesh

skonto · 2024-06-18T09:44:32Z

/test istio-latest-no-mesh

skonto · 2024-06-18T12:19:52Z

/test istio-latest-no-mesh

knative-prow · 2024-06-18T13:21:47Z

@skonto: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
istio-latest-no-mesh_serving_main	`fbf67dd`	link	true	`/test istio-latest-no-mesh`

Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

skonto · 2024-06-18T15:51:06Z

Last run here shows that:

After restarting we have,

...
chaosduck-f45fb7b47-l85xb              1/1     Running   0          28m
controller-5bfdc8f748-7nx6j            1/1     Running   0          3s
controller-5bfdc8f748-dfgds            1/1     Running   0          3s
controller-5bfdc8f748-mn922            1/1     Running   0          3s

Controller pods have 9 reconcilers as expected:

chaosduck kills one of the controllers

However before that and before we update the cm chaos is also deleting pods leaving leases not cleaned up:

controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.00-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.01-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.02-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.03-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.04-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.05-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.06-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.07-of-10  
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.08-of-10  controller-5bfdc8f748-rwwfx_71275a6d-e52c-4375-a4ed-a011c97c9dfe
controller.knative.dev.serving.pkg.reconciler.certificate.reconciler.09-of-10

The controller pod created before the ones above will run leader election on the certificate reconciler as it probably runs before cm update:

However soon enough it stops leading:

As chaos kicks in again:

However not all leases are released since the last update fails with:

skonto · 2024-06-18T16:11:31Z

It seems go client does not clean up everything.
From the timestamps it is shown that controller-5bfdc8f748-rwwfx starts before controller-5bfdc8f748-7nx6j due to chaos and uses the old cm settings.

skonto · 2024-06-18T16:44:03Z

A lot of controller pods are being created due to chaos:

Full list downloaded-logs-20240618-194319.csv.

knative-prow-robot · 2024-06-27T06:37:44Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

test for controller ha

39aecf0

knative-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2024

knative-prow bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 10, 2024

knative-prow bot requested review from izabelacg and ReToCode June 10, 2024 16:08

skonto added 2 commits June 11, 2024 10:53

print lease holder ids

036974c

fix lease printing

facb70b

skonto commented Jun 12, 2024

View reviewed changes

fix leases

6d3b0fa

skonto mentioned this pull request Jun 15, 2024

TestControllerHA is flakey #15238

Open

less noise

fbf67dd

skonto mentioned this pull request Jun 18, 2024

Lease update in release fails during graceful shutdown kubernetes/client-go#1362

Open

skonto mentioned this pull request Jun 26, 2024

Clean up certificate leases #15359

Open

knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] [test-only] Test for controller ha #15321

[wip] [test-only] Test for controller ha #15321

skonto commented Jun 10, 2024

knative-prow bot commented Jun 10, 2024

codecov bot commented Jun 10, 2024 •

edited

Loading

skonto commented Jun 10, 2024

skonto commented Jun 11, 2024 •

edited

Loading

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 12, 2024

skonto commented Jun 12, 2024

skonto Jun 12, 2024 •

edited

Loading

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024 •

edited

Loading

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024

skonto commented Jun 15, 2024

skonto commented Jun 15, 2024

skonto commented Jun 15, 2024 •

edited

Loading

skonto commented Jun 18, 2024

skonto commented Jun 18, 2024

skonto commented Jun 18, 2024

knative-prow bot commented Jun 18, 2024

skonto commented Jun 18, 2024 •

edited

Loading

skonto commented Jun 18, 2024 •

edited

Loading

skonto commented Jun 18, 2024

knative-prow-robot commented Jun 27, 2024

[wip] [test-only] Test for controller ha #15321

Are you sure you want to change the base?

[wip] [test-only] Test for controller ha #15321

Conversation

skonto commented Jun 10, 2024

Proposed Changes

knative-prow bot commented Jun 10, 2024

codecov bot commented Jun 10, 2024 • edited Loading

Codecov Report

skonto commented Jun 10, 2024

skonto commented Jun 11, 2024 • edited Loading

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 11, 2024

skonto commented Jun 12, 2024

skonto commented Jun 12, 2024

skonto Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024 • edited Loading

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024

skonto commented Jun 14, 2024

skonto commented Jun 15, 2024

skonto commented Jun 15, 2024

skonto commented Jun 15, 2024 • edited Loading

skonto commented Jun 18, 2024

skonto commented Jun 18, 2024

skonto commented Jun 18, 2024

knative-prow bot commented Jun 18, 2024

skonto commented Jun 18, 2024 • edited Loading

skonto commented Jun 18, 2024 • edited Loading

skonto commented Jun 18, 2024

knative-prow-robot commented Jun 27, 2024

codecov bot commented Jun 10, 2024 •

edited

Loading

skonto commented Jun 11, 2024 •

edited

Loading

skonto Jun 12, 2024 •

edited

Loading

skonto commented Jun 14, 2024 •

edited

Loading

skonto commented Jun 15, 2024 •

edited

Loading

skonto commented Jun 18, 2024 •

edited

Loading

skonto commented Jun 18, 2024 •

edited

Loading