[BUG] Not all Pods get restartet after Secret Change

[teimyBr]

Describe the bug
We have like 20 Deployment in our cluster.
all have on the Deployment the Annotations: reloader.stakater.com/auto: true

This all are AKHQ Deployment with Kafka Secrets.

Every 5 Days the Secrets get changed. (at the same time)

Sometimes:
Only 17/18 get restartet by reloader.

level=info msg="Changes detected in 'root-ca-cert-truststore' of type 'SECRET' in namespace 'test1', Updated 'akhq' of type 'Deployment' in namespace 'test1'"

Log look like this without any error.

To Reproduce
Who can i debug this deeper ?

Expected behavior
All pods get restartet

Environment

• Operator Version: v1.0.108
• Kubernetes/OpenShift Version: 1.28.6 Plain Vanilla Kubernetes

[teimyBr]

More: He logs level=info msg="Changes detected in 'root-ca-cert-truststore' of type 'SECRET' in namespace 'test1', Updated 'akhq' of type 'Deployment' in namespace 'test1'"

But sometimes the pod doesnt restartet

so 17 or 18 have restartet and 1-2 NOT

But the Log said he restartet all 20.

There is no error log that something has failed.

[theBNT]

Is the restarting done in a fire-and-request approach and if the API server has issues, they are lost or is there some ACK/retry involved?

[MuneebAijaz]

The pods which are not restarted, are they the same ones everytime or random?

[teimyBr]

Very Random. We watch this over the last 3-4 weeks. sometime this is deployment 17 then next time deployment 3.
Never ever the same.

Tried it also with latest version there is it still there

[aswindevs]

Facing the same issue.
Sometimes not all deployments gets rolled out. We have 56 deployments in total.

[MuneebAijaz]

any more information about what values are being used to install Reloader?
and are all deployments backed by any CD tool, if yes, is there a possibility of CD tool and Reloader clashing in updating Deployments?

[shameemshah]

+1 Facing the same issue.

[teimyBr]

We are using reloader helm chart.

Chart.yaml

apiVersion: v2
name: reloader
version: 0.0.0
dependencies:
- name: reloader
  version: 1.0.114
  repository: https://stakater.github.io/stakater-charts

values.yaml

reloader:
  reloader:
    deployment:
      resources:
        limits:
          cpu: 500m
          memory: 256Mi
        requests:
          cpu: 10m
          memory: 128Mi