workflow status not updated timely

[heidongxianhua]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

when I check the workflow, the pod of last step of this workflow has completed, but the wf status is pending.

and the pod status:

and according the pod logs, this pod finished just need 1 min, and there are 1700 running workflow total. And according the logs from controller, it is maybe caused by the max parallelism limit, and the parallelism value is 6000 in configmap.

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

none, cannot reproduce stably.

Logs from the workflow controller

time="2024-06-24T02:29:55.215Z" level=info msg="Processing workflow" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.220Z" level=info msg="Task-result reconciliation" namespace=bytevs numObjs=0 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.220Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.220Z" level=info msg="node unchanged" namespace=bytevs nodeID=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.223Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.227Z" level=info msg="Workflow step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-254110657 not yet completed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.227Z" level=info msg="TaskSet Reconciliation" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:29:55.227Z" level=info msg=reconcileAgentPod namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.850Z" level=info msg="Processing workflow" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.854Z" level=info msg="Task-result reconciliation" namespace=bytevs numObjs=0 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.854Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.854Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.854Z" level=info msg="node changed" namespace=bytevs new.message= new.phase=Succeeded new.progress=0/1 nodeID=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 old.message= old.phase=Running old.progress=0/1 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.854Z" level=info msg="setting workflow.outputs.parameters.step-vina-docking: 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockvinadocking-k92s7-7f1z5-1142978766'" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.855Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=info msg="Step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-254110657 successful" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-254110657 phase Running -> Succeeded" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-254110657 finished: 2024-06-24 02:32:34.85803606 +0000 UTC" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=info msg="StepGroup node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 initialized Running" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.858Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:34.860Z" level=info msg="Pod node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 initialized Pending" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.203Z" level=info msg="Created pod: posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l[2].step-ligand-prepare (posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354)" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.203Z" level=info msg="Workflow step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 not yet completed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.203Z" level=info msg="TaskSet Reconciliation" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.203Z" level=info msg=reconcileAgentPod namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.256Z" level=info msg="Workflow update successful" namespace=bytevs phase=Running resourceVersion=1339611002 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:35.262Z" level=info msg="cleaning up pod" action=labelPodCompleted key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockvinadocking-k92s7-7f1z5-1142978766/labelPodCompleted
time="2024-06-24T02:32:44.953Z" level=info msg="Processing workflow" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.956Z" level=info msg="Task-result reconciliation" namespace=bytevs numObjs=0 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.956Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.957Z" level=info msg="node changed" namespace=bytevs new.message= new.phase=Running new.progress=0/1 nodeID=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 old.message= old.phase=Pending old.progress=0/1 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.958Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.958Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.959Z" level=info msg="Workflow step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 not yet completed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.960Z" level=info msg="TaskSet Reconciliation" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:44.960Z" level=info msg=reconcileAgentPod namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:32:45.322Z" level=info msg="Workflow update successful" namespace=bytevs phase=Running resourceVersion=1339612836 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.805Z" level=info msg="Processing workflow" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.808Z" level=info msg="Task-result reconciliation" namespace=bytevs numObjs=0 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.809Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.809Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.809Z" level=info msg="node changed" namespace=bytevs new.message= new.phase=Succeeded new.progress=0/1 nodeID=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 old.message= old.phase=Running old.progress=0/1 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.809Z" level=info msg="setting workflow.outputs.parameters.step-ligand-prepare: 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354'" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.810Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.810Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=info msg="Step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 successful" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 phase Running -> Succeeded" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 finished: 2024-06-24 02:33:30.812188663 +0000 UTC" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=info msg="StepGroup node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1260370439 initialized Running" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.812Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:30.814Z" level=info msg="Pod node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4148759088 initialized Pending" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.016Z" level=info msg="Created pod: posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l[3].step-minimize-cpp (posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockminimize-v8w7l-ot9ur-4148759088)" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.017Z" level=info msg="Workflow step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1260370439 not yet completed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.017Z" level=info msg="TaskSet Reconciliation" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.017Z" level=info msg=reconcileAgentPod namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.125Z" level=info msg="Workflow update successful" namespace=bytevs phase=Running resourceVersion=1339623623 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:31.132Z" level=info msg="cleaning up pod" action=labelPodCompleted key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354/labelPodCompleted
time="2024-06-24T02:33:41.106Z" level=info msg="Processing workflow" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.109Z" level=info msg="Task-result reconciliation" namespace=bytevs numObjs=0 workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.110Z" level=error msg="was unable to obtain node for posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4148759088" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.110Z" level=info msg="Workflow pod is missing" namespace=bytevs nodeName="posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l[2].step-ligand-prepare" nodePhase=Running recentlyStarted=false workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.110Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 phase Running -> Error" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.110Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 message: pod deleted" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.110Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 finished: 2024-06-24 02:33:41.110077329 +0000 UTC" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.111Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-3406984913]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.111Z" level=info msg="SG Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766 are [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1142978766]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Step group node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 deemed failed: child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 phase Running -> Failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 message: child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 finished: 2024-06-24 02:33:41.113189699 +0000 UTC" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="step group posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1327628010 was unsuccessful: child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354 is [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Outbound nodes of posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l is [posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354]" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l phase Running -> Failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l message: child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l finished: 2024-06-24 02:33:41.113258001 +0000 UTC" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="TaskSet Reconciliation" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg=reconcileAgentPod namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Updated phase Running -> Failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Updated message  -> child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Marking workflow completed" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.113Z" level=info msg="Marking workflow as pending archiving" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.119Z" level=info msg="cleaning up pod" action=deletePod key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-1340600742-agent/deletePod
time="2024-06-24T02:33:41.218Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.218Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:41.321Z" level=info msg="Failed to re-apply update" error="must never update completed node posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354" namespace=bytevs workflow=posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:33:51.245Z" level=info msg="Workflow processing has been postponed due to max parallelism limit" key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T02:34:26.603Z" level=info msg="Workflow processing has been postponed due to max parallelism limit" key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
E0624 02:35:30.103821       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l.17dbd0094b9528a1", GenerateName:"", Namespace:"bytevs", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"bytevs", Name:"posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l", UID:"8f6dff64-4724-42b8-bff1-095313c50b02", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"1339612836", FieldPath:""}, Reason:"WorkflowFailed", Message:"child 'posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-4106583354' failed", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:time.Date(2024, time.June, 24, 2, 33, 41, 113325729, time.Local), LastTimestamp:time.Date(2024, time.June, 24, 2, 33, 41, 113325729, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Too many requests: limit reached on type User for key system:serviceaccount:argo:argo' (will not retry!)
time="2024-06-24T02:54:26.604Z" level=info msg="Workflow processing has been postponed due to max parallelism limit" key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T03:14:26.606Z" level=info msg="Workflow processing has been postponed due to max parallelism limit" key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l
time="2024-06-24T03:34:26.607Z" level=info msg="Workflow processing has been postponed due to max parallelism limit" key=bytevs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l

Logs from in your workflow's wait container

time="2024-06-24T02:32:24.768Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/serverless_output"
time="2024-06-24T02:32:24.768Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockvinadocking-k92s7-7f1z5-1142978766/main.log"
time="2024-06-24T02:32:24.768Z" level=info msg="Creating minio client using static credentials" endpoint=tos-s3-cn-beijing-inner.ivolces.com
time="2024-06-24T02:32:24.770Z" level=info msg="Saving file to s3" bucket=ai4s endpoint=tos-s3-cn-beijing-inner.ivolces.com key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockvinadocking-k92s7-7f1z5-1142978766/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:32:24.780Z" level=info msg="Save artifact" artifactName=main-logs duration=12.087666ms error="<nil>" key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockvinadocking-k92s7-7f1z5-1142978766/main.log
time="2024-06-24T02:32:24.780Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:32:24.780Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-24T02:32:24.786Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:bytevs:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"bytevs\""
time="2024-06-24T02:32:24.814Z" level=info msg="Alloc=11694 TotalAlloc=25513 Sys=50045 NumGC=7 Goroutines=12"
time="2024-06-24T02:32:24.814Z" level=info msg="Deadline monitor stopped"
time="2024-06-24T02:33:20.939Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/serverless_output"
time="2024-06-24T02:33:20.939Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354/main.log"
time="2024-06-24T02:33:20.939Z" level=info msg="Creating minio client using static credentials" endpoint=tos-s3-cn-beijing-inner.ivolces.com
time="2024-06-24T02:33:20.941Z" level=info msg="Saving file to s3" bucket=ai4s endpoint=tos-s3-cn-beijing-inner.ivolces.com key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:33:20.951Z" level=info msg="Save artifact" artifactName=main-logs duration=12.236806ms error="<nil>" key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockligandprepare-bxj9c-2i67o-4106583354/main.log
time="2024-06-24T02:33:20.951Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:33:20.951Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-24T02:33:20.958Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:bytevs:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"bytevs\""
time="2024-06-24T02:33:20.983Z" level=info msg="Alloc=13746 TotalAlloc=25184 Sys=44925 NumGC=5 Goroutines=12"
time="2024-06-24T02:33:20.983Z" level=info msg="Deadline monitor stopped"
time="2024-06-24T02:34:16.615Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/serverless_output"
time="2024-06-24T02:34:16.615Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockminimize-v8w7l-ot9ur-4148759088/main.log"
time="2024-06-24T02:34:16.615Z" level=info msg="Creating minio client using static credentials" endpoint=tos-s3-cn-beijing-inner.ivolces.com
time="2024-06-24T02:34:16.617Z" level=info msg="Saving file to s3" bucket=ai4s endpoint=tos-s3-cn-beijing-inner.ivolces.com key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockminimize-v8w7l-ot9ur-4148759088/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:34:16.631Z" level=info msg="Save artifact" artifactName=main-logs duration=16.214241ms error="<nil>" key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockminimize-v8w7l-ot9ur-4148759088/main.log
time="2024-06-24T02:34:16.631Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:34:16.631Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-24T02:34:16.639Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:bytevs:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"bytevs\""
time="2024-06-24T02:34:16.666Z" level=info msg="Alloc=18266 TotalAlloc=25319 Sys=54653 NumGC=5 Goroutines=12"
time="2024-06-24T02:34:16.666Z" level=info msg="Deadline monitor stopped"
time="2024-06-24T02:27:25.435Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockpreparereceptor-fn1xk-sb2u4-3406984913/main.log"
time="2024-06-24T02:27:25.435Z" level=info msg="Creating minio client using static credentials" endpoint=tos-s3-cn-beijing-inner.ivolces.com
time="2024-06-24T02:27:25.443Z" level=info msg="Saving file to s3" bucket=ai4s endpoint=tos-s3-cn-beijing-inner.ivolces.com key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockpreparereceptor-fn1xk-sb2u4-3406984913/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:27:25.457Z" level=info msg="Save artifact" artifactName=main-logs duration=21.851559ms error="<nil>" key=argo_logs/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l/posebuster-bytedock-cpp-eval-7mmh-zjy-bmv8l-bytedockpreparereceptor-fn1xk-sb2u4-3406984913/main.log
time="2024-06-24T02:27:25.457Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-24T02:27:25.457Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-24T02:27:25.464Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:bytevs:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"bytevs\""
time="2024-06-24T02:27:25.489Z" level=info msg="Alloc=10582 TotalAlloc=19252 Sys=45181 NumGC=5 Goroutines=10"
time="2024-06-24T02:27:25.489Z" level=info msg="Deadline monitor stopped"
time="2024-06-24T02:27:25.489Z" level=info msg="stopping progress monitor (context done)" error="context canceled"