[Bug] fix ArrayNode state's TaskPhase reset

[pvditt]

Tracking issue

Why are the changes needed?

When a NodePhase change is detected, the current implementation resets the TaskPhaseVersion to 0 prior to emitting the event. This would create a duplicate event since the TaskPhase isn't also updated causing for the event to not get emitted/persisted to admin.

What changes were proposed in this pull request?

• Bump the task phase version after emitting the event when a NodePhase change is detected in the ArrayNode handler.

---

We opt to do this instead of bumping the task phase as that can cause issues.
Example issue:

• updating the task phase to a terminal phase and then emitting that event to admin would set that task execution to a terminal state
• admin does not persist follow up event when a task execution is already in a terminal state
• the next propeller loop that handles cleanup wouldn't be able to persist new task events such as aborting subnodes on faliure cleanup.

Also this seems to be consistent with letting the proceeding propeller loop handling the next state.

How was this patch tested?

Ran through different failing and succeeding scenarios and ensured that the subnodes phases transitioned to the correct terminal phase.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.07%. Comparing base (25c3596) to head (d47547e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5451   +/-   ##
=======================================
  Coverage   61.07%   61.07%           
=======================================
  Files         793      793           
  Lines       51226    51226           
=======================================
  Hits        31288    31288           
  Misses      17062    17062           
  Partials     2876     2876           

Flag	Coverage Δ
unittests-datacatalog	69.31% <ø> (ø)
unittests-flyteadmin	58.90% <ø> (ø)
unittests-flytecopilot	17.79% <ø> (ø)
unittests-flytectl	67.97% <ø> (ø)
unittests-flyteidl	79.04% <ø> (ø)
unittests-flyteplugins	61.94% <ø> (ø)
unittests-flytepropeller	57.32% <100.00%> (ø)
unittests-flytestdlib	65.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hamersaw]

Doesn't setting this afterwards cause problems with admin requiring incremental values?

[pvditt]

The scenario of interest would be if we would lose some eventing data if we have incrementPhase = False and currentArrayNodePhase != arrayNodeState.Phase + have a previous event emitted with the same TaskPhase and TaskPhaseVersion.

We set incrementTaskPhaseVersion = True if there's a subnode phase change. The arrayNodeState.Phase is updated in 3 different places: arrayNodeState.Phase = v1alpha1.ArrayNodePhaseFailing, arrayNodeState.Phase = v1alpha1.ArrayNodePhaseSucceeding, and arrayNodeState.Phase = v1alpha1.ArrayNodePhaseExecuting.

For arrayNodeState.Phase = v1alpha1.ArrayNodePhaseFailing and arrayNodeState.Phase = v1alpha1.ArrayNodePhaseSucceeding, there would have to be subnode phase change so we couldn't have a scenario where incrementPhase = False and currentArrayNodePhase != arrayNodeState.Phase.

For arrayNodeState.Phase = v1alpha1.ArrayNodePhaseExecuting, we shouldn't have a previous event emitted with the same TaskPhase and TaskPhaseVersion as we should only be in the v1alpha1.ArrayNodePhaseNone phase for the first pass through.

[pvditt]

The proceeding loops would have a new TaskPhase as well.