Velero is not honouring default keep-latest-maintenance-jobs

[veerendra2]

What steps did you take and what happened:

Deploy velero server with default options(i.e don't set --keep-latest-maintenance-jobs) and uploader type kopia. Create backups with --snapshot-move-data like below

velero backup create veerendra-backup-8h \
  --include-namespaces veerendra-test \
  --snapshot-move-data \
  --include-resources persistentvolumeclaims \
  --csi-snapshot-timeout 8h \
  --parallel-files-upload 64

From help, the default value for --keep-latest-maintenance-jobs is 3

$ ./velero server --help | grep keep-latest-maintenance-jobs
      --keep-latest-maintenance-jobs int                    Number of latest maintenance jobs to keep each repository. Optional. (default 3)

When --keep-latest-maintenance-jobs not set, there should be 3 job per backup repository, but

auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185841pp9kg   0/1     Completed   0                12h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185877dvtgh   0/1     Completed   0                11h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185934zmt2l   0/1     Completed   0                10h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718584527745kvm9   0/1     Completed   0                12h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718588388245th6g   0/1     Completed   0                11h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-171859205501l2nwj   0/1     Completed   0                10h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171858297hdtlm   0/1     Completed   0                13h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-1718586572jzg8   0/1     Completed   0                12h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859017wg47g   0/1     Completed   0                11h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt   1/1     Running     0                10h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185845006gwgvq   0/1     Completed   0                12h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185883822qls6c   0/1     Completed   0                11h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185935585bjpvd   0/1     Completed   0                10h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185844826xwxs5   0/1     Completed   0                12h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185880822fz45k   0/1     Completed   0                11h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-171859358179f4c4   0/1     Completed   0                10h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718584507lzxjd   0/1     Completed   0                12h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718588394qpdvr   0/1     Completed   0                11h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718592161gb4h2   0/1     Completed   0                10h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-1718583892q2ns   0/1     Completed   0                12h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171858778rx9km   0/1     Completed   0                11h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171859346hd5r9   0/1     Completed   0                10h
repo-maintain-job-1718583275574-fjvdx                             0/1     Completed   0                13h
repo-maintain-job-1718583575574-9dxbj                             0/1     Completed   0                12h
repo-maintain-job-1718583587497-59dzn                             0/1     Completed   0                12h
repo-maintain-job-1718583595539-c49bk                             0/1     Completed   0                12h
repo-maintain-job-1718583875575-xsdq4                             0/1     Completed   0                12h
repo-maintain-job-1718583881845-9jwsm                             0/1     Completed   0                12h
repo-maintain-job-1718583886866-btw8n                             0/1     Completed   0                12h
repo-maintain-job-1718584475577-ms46g                             0/1     Completed   0                12h
repo-maintain-job-1718584488615-ktw7z                             0/1     Completed   0                12h
repo-maintain-job-1718584494633-5sjqv                             0/1     Completed   0                12h
repo-maintain-job-1718584513718-vkh4h                             0/1     Completed   0                12h
repo-maintain-job-1718584533746-6gq8g                             0/1     Completed   0                12h
repo-maintain-job-1718584540761-ccmv2                             0/1     Completed   0                12h
repo-maintain-job-1718586875584-g4d6q                             0/1     Completed   0                12h
repo-maintain-job-1718587175585-8jm8l                             0/1     Completed   0                11h
repo-maintain-job-1718587475585-tzsnf                             0/1     Completed   0                11h
repo-maintain-job-1718587483390-rvfpx                             0/1     Completed   0                11h
repo-maintain-job-1718587489403-bpfkg                             0/1     Completed   0                11h
repo-maintain-job-1718587496427-mk87t                             0/1     Completed   0                11h
repo-maintain-job-1718587775585-whwg8                             0/1     Completed   0                11h
repo-maintain-job-1718588075586-7l2l9                             0/1     Completed   0                11h
repo-maintain-job-1718588097309-kgjw2                             0/1     Completed   0                11h
repo-maintain-job-1718588375587-8ddvp                             0/1     Completed   0                11h
repo-maintain-job-1718588400297-bp8bb                             0/1     Completed   0                11h
repo-maintain-job-1718588415359-6g2sw                             0/1     Completed   0                11h
repo-maintain-job-1718588423385-lkddt                             0/1     Completed   0                11h
repo-maintain-job-1718590475592-9wdgv                             0/1     Completed   0                11h
repo-maintain-job-1718590775593-kls5g                             0/1     Completed   0                10h
repo-maintain-job-1718592174416-4298z                             0/1     Completed   0                10h
repo-maintain-job-1718593379857-kz96d                             0/1     Completed   0                10h
repo-maintain-job-1718593386910-6fsdg                             0/1     Completed   0                10h
repo-maintain-job-1718593396937-55vpc                             0/1     Completed   0                10h
repo-maintain-job-1718593435141-l4kmt                             0/1     Completed   0                10h
repo-maintain-job-1718593443182-wjv6t                             0/1     Completed   0                10h
repo-maintain-job-1718593475326-ftj4r                             0/1     Completed   0                10h
repo-maintain-job-1718593548635-w8m5t                             0/1     Completed   0                10h
repo-maintain-job-1718593573752-b9xb4                             0/1     Completed   0                10h
repo-maintain-job-1718593587806-nr2sc                             0/1     Completed   0                10h
repo-maintain-job-1718593596321-87w5k                             0/1     Completed   0                10h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718585675pbgsm   0/1     Completed   0                12h
search-backup-prod-de1-kopia-52bbc-maintain-job-17185892756kpff   0/1     Completed   0                11h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718593601bh4vp   0/1     Completed   0                10h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-171858625mdpz   0/1     Completed   0                12h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185898g2z2p   0/1     Completed   0                11h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185935dsdnl   0/1     Completed   0                10h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858452rqqhf   0/1     Completed   0                12h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858840dmvvc   0/1     Completed   0                11h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171859337m9jld   0/1     Completed   0                10h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858597558c469z   0/1     Completed   0                12h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858957558rtlkb   0/1     Completed   0                11h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-1718593609917bkf7   0/1     Completed   0                10h

What did you expect to happen:
There should be only 3 maintain-job pods per backup repositories

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

velero-server logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8

Environment:
velero-server

          command:
            - /velero
          args:
            - server
            ### Flags
            - --uploader-type=kopia
            - --default-backup-storage-location=backup
            - --log-format=json
            - --keep-latest-maintenance-jobs=1
            ### Global Flags
            - --features=EnableCSI
            - --default-backup-ttl=${velero_schedule_ttl}
            - --resource-timeout=60m
            - --terminating-resource-timeout=60m

node-agent

      containers:
        - args:
            - node-agent
            - server
            - --data-mover-prepare-timeout=60m
            - --resource-timeout=60m
            - --features=EnableCSI
            - --log-format=json
          command:
            - /velero

• Velero version (use velero version): Server v1.13.2
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): Server Version: v1.28.5
• Kubernetes installer & version:
• Cloud provider or hardware configuration: Azure AKS
• OS (e.g. from /etc/os-release): Ubuntu

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

Please share the velero log bundle from the time when the maintenance jobs were created to the latest.

[veerendra2]

velero-server logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8

[Lyndon-Li]

@veerendra2 We need the entire log bundle, you can get it by velero debug command.

[Lyndon-Li]

I see there is one running job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt lasted 10h, is that expected? If not, what happened to that job?
I think that job blocked deletion of other jobs

[Lyndon-Li]

@veerendra2
Please help to collect the log for job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt and we need to check what happened to the job and why it kept running for 10h

[Lyndon-Li]

Moreover, you need to keep the Velero server and client in the same version, otherwise, you may see many weird issues due to failures on resource operations since the CRD version mismatch.
As we can see here, the version info seems not correct:
Velero version (use velero version): Server v1.13.2

Therefore, if the version mismatch, please retry it after fixing the version problem.

[veerendra2]

@Lyndon-Li Sorry, I lost notifications. The velero-server is upgraded and logs are gone. This velero debug never worked for me

$ k get deployment -n velero
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
velero   1/1     1            1           357d

$ velero debug
An error occurred: velero deployment does not exist in namespace: velero

[veerendra2]

I will remove --keep-latest-maintenance-jobs in our dev cluster today and keep it for this weekend. On next Monday, I will share the logs.

[veerendra2]

@Lyndon-Li Here is the info

I see there is one running job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt lasted 10h, is that expected?

No.(But, as I mentioned above, logs are gone. I'm attaching new logs below, please check)

Pods

$ k get pods
NAME                                                              READY   STATUS      RESTARTS        AGE
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-1719209077nwg   0/1     Completed   0               137m
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-1719212984k8q   0/1     Completed   0               73m
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-17192167pfgkc   0/1     Completed   0               8m53s
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171920721bpp86   0/1     Completed   0               168m
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171921107xnktn   0/1     Completed   0               103m
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171921471495jt   0/1     Completed   0               43m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719208970921z4f55   0/1     Completed   0               139m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719212584309vms29   0/1     Completed   0               78m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719216477608b6cpt   0/1     Completed   0               13m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-1719205973ss2qj   0/1     Completed   0               3h8m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-17192098702w8tz   0/1     Completed   0               124m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-1719213473pf596   0/1     Completed   0               63m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-17192059ngb8p   0/1     Completed   0               3h8m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-17192098fgb6c   0/1     Completed   0               123m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-1719213428s8k   0/1     Completed   0               63m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-1719207197r6vrk   0/1     Completed   0               168m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-17192108099qqhd   0/1     Completed   0               108m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-1719214712msr7h   0/1     Completed   0               43m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171920598522wf8gb   0/1     Completed   0               3h8m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171920988447nk7hx   0/1     Completed   0               123m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171921348742xxhnw   0/1     Completed   0               63m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719207770g9dg2   0/1     Completed   0               159m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719211377nm96w   0/1     Completed   0               98m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719215270g5nvw   0/1     Completed   0               34m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192059rdf9j   0/1     Completed   0               3h8m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192098gz684   0/1     Completed   0               123m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192134k7pcd   0/1     Completed   0               63m
node-agent-256mw                                                  1/1     Running     0               2d22h
node-agent-2whq8                                                  1/1     Running     0               3d20h
node-agent-5dxdq                                                  1/1     Running     0               3d20h
node-agent-6grrp                                                  1/1     Running     0               3d20h
node-agent-7mjg6                                                  1/1     Running     0               3d20h
node-agent-bv8qv                                                  1/1     Running     0               3d20h
node-agent-hz4bw                                                  1/1     Running     0               3d20h
node-agent-kt7cr                                                  1/1     Running     0               3d20h
node-agent-kz8l9                                                  1/1     Running     0               3d20h
node-agent-mctdc                                                  1/1     Running     0               3d20h
node-agent-mmbmb                                                  1/1     Running     0               3d20h
node-agent-n2v96                                                  1/1     Running     0               3d20h
node-agent-q7n6s                                                  1/1     Running     0               3d20h
node-agent-t662b                                                  1/1     Running     0               3d20h
node-agent-xbt6b                                                  1/1     Running     0               3d20h
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192074794lkckk   0/1     Completed   0               163m
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192113912h7svd   0/1     Completed   0               98m
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192152857g7ftt   0/1     Completed   0               33m
pacman-backup-dev-de1-restic-q8mmf-maintain-job-1718877154qwr47   0/1     Completed   0               3d22h
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719207485450rsc7h   0/1     Completed   0               163m
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719211370929nc6zw   0/1     Completed   0               99m
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719214972980rt77w   0/1     Completed   0               38m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-17192072064j59qd   0/1     Completed   0               168m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-1719210822422lph   0/1     Completed   0               108m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-17192147061x5vbz   0/1     Completed   0               43m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719206886n974t   0/1     Completed   0               173m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719210770hb97v   0/1     Completed   0               109m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719214373zj4nh   0/1     Completed   0               48m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192060nctd8   0/1     Completed   0               3h8m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192099gmpb4   0/1     Completed   0               123m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192137wzkdn   0/1     Completed   0               59m
repo-maintain-job-1719206270915-2vft4                             0/1     Completed   0               3h4m
repo-maintain-job-1719206279558-kpr6m                             0/1     Completed   0               3h3m
repo-maintain-job-1719206285243-qgn24                             0/1     Completed   0               3h3m
repo-maintain-job-1719206873465-h72vf                             0/1     Completed   0               173m
repo-maintain-job-1719206880264-bn649                             0/1     Completed   0               173m
repo-maintain-job-1719206892334-k4lzz                             0/1     Completed   0               173m
repo-maintain-job-1719207173175-gm87k                             0/1     Completed   0               168m
repo-maintain-job-1719207179310-nrj4f                             0/1     Completed   0               168m
repo-maintain-job-1719207185326-t4c78                             0/1     Completed   0               168m
repo-maintain-job-1719207191363-9gjvm                             0/1     Completed   0               168m
repo-maintain-job-1719207470917-qk2v2                             0/1     Completed   0               164m
repo-maintain-job-1719207777445-r59sg                             0/1     Completed   0               158m
repo-maintain-job-1719208073179-sjzpr                             0/1     Completed   0               153m
repo-maintain-job-1719208370920-9f8rb                             0/1     Completed   0               149m
repo-maintain-job-1719208672868-m45l7                             0/1     Completed   0               143m
repo-maintain-job-1719208977856-tp5r6                             0/1     Completed   0               138m
repo-maintain-job-1719208983888-dhc7x                             0/1     Completed   0               138m
repo-maintain-job-1719208989916-g2ngg                             0/1     Completed   0               138m
repo-maintain-job-1719209004987-ldzvg                             0/1     Completed   0               138m
repo-maintain-job-1719209011005-mv9p9                             0/1     Completed   0               138m
repo-maintain-job-1719209022060-gf6tx                             0/1     Completed   0               138m
repo-maintain-job-1719209028072-ddsws                             0/1     Completed   0               138m
repo-maintain-job-1719209042143-f2phj                             0/1     Completed   0               137m
repo-maintain-job-1719209273427-kd97d                             0/1     Completed   0               133m
repo-maintain-job-1719209280087-8gkjf                             0/1     Completed   0               133m
repo-maintain-job-1719209908574-z7sxn                             0/1     Completed   0               123m
repo-maintain-job-1719210173336-v79t6                             0/1     Completed   0               118m
repo-maintain-job-1719210179714-qvcxf                             0/1     Completed   0               118m
repo-maintain-job-1719210473506-8sq2p                             0/1     Completed   0               113m
repo-maintain-job-1719210777234-gj25r                             0/1     Completed   0               108m
repo-maintain-job-1719210784436-xccc2                             0/1     Completed   0               108m
repo-maintain-job-1719210790243-7l4xr                             0/1     Completed   0               108m
repo-maintain-job-1719210796299-qsxxh                             0/1     Completed   0               108m
repo-maintain-job-1719210803307-klmf5                             0/1     Completed   0               108m
repo-maintain-job-1719210816386-tg5lv                             0/1     Completed   0               108m
repo-maintain-job-1719211080497-92kcn                             0/1     Completed   0               103m
repo-maintain-job-1719211383817-7rs7q                             0/1     Completed   0               98m
repo-maintain-job-1719211970930-xnzd9                             0/1     Completed   0               89m
repo-maintain-job-1719211978495-qssmt                             0/1     Completed   0               88m
repo-maintain-job-1719212570931-rvs4b                             0/1     Completed   0               79m
repo-maintain-job-1719212578228-xmhj6                             0/1     Completed   0               78m
repo-maintain-job-1719212591336-7mg8h                             0/1     Completed   0               78m
repo-maintain-job-1719212597369-m2vl7                             0/1     Completed   0               78m
repo-maintain-job-1719212872898-vq5jn                             0/1     Completed   0               73m
repo-maintain-job-1719212879622-x89cs                             0/1     Completed   0               73m
repo-maintain-job-1719212890695-jcd6f                             0/1     Completed   0               73m
repo-maintain-job-1719212895758-bljjw                             0/1     Completed   0               73m
repo-maintain-job-1719212907794-cl4ql                             0/1     Completed   0               73m
repo-maintain-job-1719212914813-sp8fb                             0/1     Completed   0               73m
repo-maintain-job-1719212920871-tz62j                             0/1     Completed   0               73m
repo-maintain-job-1719213777779-d5vbw                             0/1     Completed   0               58m
repo-maintain-job-1719213786316-g698w                             0/1     Completed   0               58m
repo-maintain-job-1719213792884-fmzxq                             0/1     Completed   0               58m
repo-maintain-job-1719214073073-qqtcg                             0/1     Completed   0               53m
repo-maintain-job-1719214380513-bmxf2                             0/1     Completed   0               48m
repo-maintain-job-1719214673508-4ftdv                             0/1     Completed   0               43m
repo-maintain-job-1719214679939-rhqwl                             0/1     Completed   0               43m
repo-maintain-job-1719214685986-pwr2r                             0/1     Completed   0               43m
repo-maintain-job-1719214693020-dtb25                             0/1     Completed   0               43m
repo-maintain-job-1719214699105-ft7fn                             0/1     Completed   0               43m
repo-maintain-job-1719214726196-svzcb                             0/1     Completed   0               43m
repo-maintain-job-1719215277346-76mxk                             0/1     Completed   0               33m
repo-maintain-job-1719215570939-wwz7m                             0/1     Completed   0               29m
repo-maintain-job-1719215578298-rzhss                             0/1     Completed   0               28m
repo-maintain-job-1719216170941-tbkkt                             0/1     Completed   0               19m
repo-maintain-job-1719216470942-h4k5s                             0/1     Completed   0               14m
repo-maintain-job-1719216483663-pr57g                             0/1     Completed   0               13m
repo-maintain-job-1719216489668-rghrf                             0/1     Completed   0               13m
repo-maintain-job-1719216770943-lbtqs                             0/1     Completed   0               9m1s
repo-maintain-job-1719216784542-9c8zv                             0/1     Completed   0               8m47s
repo-maintain-job-1719216790579-sfsr9                             0/1     Completed   0               8m41s
repo-maintain-job-1719216796601-hvfrr                             0/1     Completed   0               8m35s
repo-maintain-job-1719216804991-k4p5q                             0/1     Completed   0               8m26s
repo-maintain-job-1719216811667-5kcsg                             0/1     Completed   0               8m20s
repo-maintain-job-1719216818757-9crlx                             0/1     Completed   0               8m13s
search-backup-dev-de1-kopia-n5khx-maintain-job-17192089979k5xfz   0/1     Completed   0               138m
search-backup-dev-de1-kopia-n5khx-maintain-job-17192126075jl9v7   0/1     Completed   0               78m
search-backup-dev-de1-kopia-n5khx-maintain-job-17192164969nwps5   0/1     Completed   0               13m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-1719209019nkmg   0/1     Completed   0               138m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-17192128878k4t   0/1     Completed   0               73m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-1719216829jr8v   0/1     Completed   0               8m7s
theming-backup-dev-de1-kopia-5d687-maintain-job-1719205997d5sqw   0/1     Completed   0               3h8m
theming-backup-dev-de1-kopia-5d687-maintain-job-1719209896cgmwb   0/1     Completed   0               123m
theming-backup-dev-de1-kopia-5d687-maintain-job-1719213500kx57j   0/1     Completed   0               63m
velero-995fb7c64-hp6m5                                            1/1     Running     1 (4h44m ago)   2d22h
velero-watchdog-28653570-xh5pz                                    0/1     Completed   0               51m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719208377640q5wgk   0/1     Completed   0               148m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719211984534ww6pt   0/1     Completed   0               88m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719215584331xfptr   0/1     Completed   0               28m

Also, you can see repo-maintain-job-* is one has more than 3 and rest are only 3 by the way!

Version

$ velero version
Client:
	Version: v1.14.0
	Git commit: -
Server:
	Version: v1.14.0

logs
velero-logs.txt

[Lyndon-Li]

@veerendra2 Could you share the entire log bundle but running velero debug command?

[veerendra2]

@Lyndon-Li I tied it again, it seems not working for me

$ velero debug
An error occurred: velero deployment does not exist in namespace: velero

$ velero debug -n velero
An error occurred: velero deployment does not exist in namespace: velero

$ k get deployment -n velero
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
velero   1/1     1            1           360d

I'm using macOS 14.5 23F79 arm64

[Lyndon-Li]

@veerendra2
Here velero debug command is trying to find velero deployment with two conditions:

1. In namespace velero
2. With label component=velero

Could you check if your installation meets these two conditions?