GitOps Design Proposal

[sriumcp]

See updated discussion all the way below...

[sriumcp]

Pull-based GitOps

Iter8 will support Pull-based GitOps using operators like Flux, ArgoCD and JenkinsX. This is illustrated in the picture below.

[sriumcp]

Common Aspects of Push-and-Pull-based GitOps

1. Iter8 can be configured with write access to environment repo -- in this case, it can directly push to the environment repo / create a PR from a branch of the environment repo.
2. Iter8 can also be configured with read access to environment repo -- in this case, it can create a PR from a fork of the environment repo.

[anggao]

1. In order for iter8 to manipulate model deployment manifests, does it require certain YAML format or it will be able to adjust traffic for arbitrary CRDs ?
2. I hope iter8 will be able to do the final promotion or rollback (write directly to the repo or as a PR) by cleaning the canary deployment section ?

[sriumcp]

1. Iter8 will make no assumption about the CRDs. It would be able to patch arbitrary YAMLs in the environment repo.
2. Yes, iter8 will be able to do the final promotion or rollback (it does so already in the initial iter8-kfserving release, where it is directly patching the InferenceService in-cluster). It will continue to due so in GitOps experiments as well through the repo.

[sriumcp]

Differences Between Push-and-Pull-based GitOps

1. In push-based GitOps, the deployment pipeline, which (at least partly) lives outside of the k8s cluster, may need god-mode enabled; i.e., credentials to access your cluster. For example, this will be the case if the k8s updates are managed through GitHub Actions.
2. In pull-based GitOps, components that lives outside of the k8s cluster do not need cluster-access credentials to perform iter8 experiments.

[sriumcp]

The same repo can be used for both app/env. Both FluxCD and ArgoCD support ability to sync specific folders / paths within the repo with the k8s cluster, so envRepo is less of a repo and more of a folder within the repo.

[sriumcp]

re: format of the deployment file -- there are two ways to teach iter8 about the format of the deployment file.

1. You can explicitly specify -- within the experiment -- how iter8 needs to adjust traffic within the given resource.
2. For specific Kubernetes stacks -- in particular, for knative and kfserving -- we are building "handlers" that will do this automatically for you at the beginning of the experiment. If you notice in the spec above, the following lines do exactly that.

start:
- library: kfserving
  task: init-experiment

[anggao]

I see, that's interesting, can a user define their own "handlers" or have a way to extend iter8 ?
Looking forward to see how this GitOps-y iter8 works in a real demo.

[sriumcp]

Yes absolutely. The handler concept is ultra powerful and extensible, and we're riding on top of it for GitOps. We will make several out-of-the-box handler capabilities available and also document how users can write their own.

[anggao]

That's cool!

[sriumcp]

References:

1. https://www.gitops.tech
2. Flagger's GitOps support: https://github.com/stefanprodan/gitops-istio

• Note: unlike Flagger, iter8 does not inline application resources into the experiment object. The issues with inlining are discussed here: https://argoproj.github.io/argo-rollouts/features/traffic-management/istio/#alternatives-considered. Iter8's experiment CRD gets over these problems since it cleanly decouples specification of your application from specification of an experiment for it.

3. ArgoRollout currently lacks GitOps support (at least, for Istio deployments): https://argoproj.github.io/argo-rollouts/features/traffic-management/istio/#integrating-with-gitops

• Note: In Flagger as well as the ArgoRollouts GitOps proposal, once the experiment is created, traffic shift is entirely hijacked by the rollout controller and does not go through the Git environment repo. This means, one of the core benefits of GitOps, specifically, easy access to and use of Git-related tooling like PRs and (manual) approvals of PRs before merges are lost during an experiment in both these systems. Iter8's GitOps design above is a significant improvement over these systems since it avoids this problem and offers a full-fledged GitOps-y solution for managing your deployed application at all times: before, during, and after the experiment.

[anggao]

Agreed, I think this is an interesting point and iter8 will make disaster recovery (DR) easier since all states are tracked by git.
(e.g. during the middle of an experiment if there is an outage, I hope iter8 will be able to continue the experiment after the recovery)

[sriumcp]

Kubecon EU just got over which means this GitOps work is now back on track and our next deliverable...

Resuming the design discussion below.

[sriumcp]

Seperation between CD pipeline segment and Iter8

Steps 0, 1 and 3

• In GitOps mode, these steps modify k8s resources in git
• In ClusterOps mode, these steps modify k8s resources in-cluster
• In both GitOps and ClusterOps mode, steps 1 and 3 can be implemented using helm or other templating mechanisms

Step 2

• Can be implemented using Iter8 tasks (example a common/exec task that uses kubectl wait). Any readiness check for K8s resources can be verified.

[sriumcp]

Reframing Iter8 quick start tutorial as per the above design.

Note: The following discussion uses helm, but alternative templating mechanisms can be used in place of helm.

Istio quick start example

Step 1

helm template bookinfo/product-page --set image={{ GITHUB.ACTIONS.INPUTS.IMAGE }},stable=false,repo={{ GITHUB.ENV.REPO }},folderPath=src/envrepo --generate-name

1. GITHUB.ACTIONS.INPUTS.IMAGE: GH actions can have inputs; image is part of the input.
2. GITHUB.ENV.REPO: GH actions have repo information available.

The above command is intended to generate the following artifacts (which can be written to Git env repo in GitOps mode and kubectl applied (or better yet, helm installed) in clusterops mode (additional values may need to be set; the command is meant to be illustrative only).

1. Create productpage-candidate.yaml
2. Create vs and dr as follows...

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  gateways:
  - mesh
  - bookinfo-gateway
  hosts:
  - productpage
  - "bookinfo.example.com"
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
        subset: productpage-stable
      weight: 100
    - destination:
        host: productpage
        port:
          number: 9080
        subset: productpage-experimental
      weight: 0
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: bookinfo
spec:
  host: productpage
  subsets:
  - labels:
      app: productpage
      version: stable
    name: productpage-stable
  - labels:
      app: productpage
      version: v2
    name: productpage-experimental

apiVersion: iter8.tools/v2alpha2
kind: Experiment
metadata:
  name: quickstart-exp-7392lasdhf
spec:
  # target identifies the service under experimentation using its fully qualified name
  target: bookinfo-iter8/productpage
  strategy:
    # this experiment will perform an A/B test
    testingPattern: A/B
    # this experiment will progressively shift traffic to the winning version
    deploymentPattern: Progressive
    actions:
      # when the experiment completes, promote the winning version using kubectl apply
      start:
      - task: common/script
        with: |
          kubectl wait --for=condition=available --timeout=600s deployment/baseline -n bookinfo-iter8
          kubectl wait --for=condition=available --timeout=600s deployment/candidate -n bookinfo-iter8
      finish:
      - task: gitops/init # drop this task for clusterops
        with:
          repo: github.com/bookinfo/productpage
          folder: env/repo/folder/path # optional; without this field, root of the repo is the default folder
          secret: ns/git-token
      - task: gitops/apply # replace gitops with clusterops for clusterops
        with:
          cmd: helm template bookinfo/product-page --set image={{ .image }},stable=true --generate-name
  criteria:
    rewards:
    # (business) reward metric to optimize in this experiment
    - metric: iter8-istio/user-engagement 
      preferredDirection: High
    objectives: # used for validating versions
    - metric: iter8-istio/mean-latency
      upperLimit: 300
    - metric: iter8-istio/error-rate
      upperLimit: "0.01"
    requestCount: iter8-istio/request-count
  duration: # product of fields determines length of the experiment
    intervalSeconds: 10
    iterationsPerLoop: 10
  versionInfo:
    # information about the app versions used in this experiment
    baseline:
      name: productpage-v1
      variables:
      - name: image 
        value: bookinfo/productpage:v1
      weightObjRef:
        apiVersion: networking.istio.io/v1beta1
        kind: VirtualService
        namespace: bookinfo-iter8
        name: bookinfo
        fieldPath: .spec.http[0].route[0].weight
    candidates:
    - name: productpage-v2
      variables:
      - name: image # used by final action if this version is the winner
        value: bookinfo/productpage:v2
      weightObjRef:
        apiVersion: networking.istio.io/v1beta1
        kind: VirtualService
        namespace: bookinfo-iter8
        name: bookinfo
        fieldPath: .spec.http[0].route[1].weight

Random string in the above experiment is generated by helm -- experiment template has

name: quickstart-exp-{{ randAlphaNum 10 }}

For Steps 2 and 3, refer to start and finish actions respectively in the experiment.

[sriumcp]

@huang195 You identified several potential problems with GitOps release automation in general in the Iter8 workgroup discussion. I am listing them below... so that we can discuss how the above proposal (and any alternative) fares with respect to those problems.

Problem 1: Problem 1 is simply stating that without iter8 writing back to the Env repo, it will not maintain GitOps guarantee, as what runs in the cluster could deviate from that of Env repo.
How the proposal addresses this problem: The proposal addresses this issue as Iter8 does write back to the repo.

Problem 2: Problem 2 says when Iter8 writes back to the Env repo, we are now introducing a race condition as now both CI and iter8 need to write to the same repo / files, which could lead to corruption.
How the proposal addresses this problem: If there are some resources that are unique to the baseline, CI never touches these (only Iter8 does). It there are some resources that are unique to the candidate, Iter8 never touches them, only CI does. If there are files that are shared (example, the Istio virtual service), both Iter8 and CI ensure correctness (this may needs to be ensured differently for different domains / applications) (in the case of the Istio VS example above, this implies, 100% of traffic is flowing to versions that are valid -- actually present in the cluster; incidentally, traffic split is also the transient state in this example).

In light of this problem, the design proposal described above provide the following three important invariants.
a. The baseline version is always the version recommended by the last completed experiment.
b. The candidate version in any active experiment (i.e., the experiment present in env repo) is the latest one.
c. Traffic split in-git and in-cluster always adds up to 100 between baseline and candidate (in git and in-cluster respectively), which ensures zero-downtime upgrades (no requests are lost during upgrades).

Problem 3 is regarding failure of handler code that changes env repo. If such code crashes in the middle, this could lead to inconsistency in env repo state. Who's responsible for recover from such failures?
How the proposal addresses this problem: Iter8 handler crashing? That's unheard of :-) Jokes apart, lets analyze this as follows.

a. What happens if one of the gitops tasks in the finish action of the above experiment crashes? The PR that would have been created by the Iter8 experiment will not be created. This does not corrupt the env repo or the cluster in anyway. It merely prevents the automated rollout from completing.
b. Assuming the user is receiving notifications for completed experiments (example through slack), they will know that experiment didn't complete. So, they can simply upgrade to the winning version in the experiment (by manually altering state in Git), and delete the experiment.
c. Of course, failure is an exception, not the common case. So, I believe this manual intervention in the automation is acceptable. If the end-user organizations have specific failure recovery patterns they use (for example, what happens if the gitops controller crashes; or the CI/CD pipeline crashes), we can explore ways to work with those patterns / tools.

Problem 4 is regarding whether or not iter8 should support both i) allow users to test every commit, and ii) allow users to test only the latest commit. My finding is it might support the former, but doesn't support the latter. Do we want to support both modes or just the former?
How the proposal addresses this problem: The proposal as documented above solves ii) and not i). It can be modified to solve i) by removing step 0, modifying step 1 so that only experiment is released and not the experimental version, and embedding the release of the experimental version as part of the experiment start action (more details needed here).

Problem 5 is regarding fine-grained sync. The use case you and Ang discussed about having iter8 sync'ing back intermediate state of an experiment has the advantage of faster recovery from a crashed experiment, but if each sync requires a separate PR, that could become a problem for human operators who need to manually merge each one.
How the proposal addresses this problem: The proposal as documented above does not sync intermediate traffic splits, only desired state at the time of version promotion. Obviously, it is possible to extend the gitops tasks to periodically write back the desired traffic split as well (if this is something users want). I do not have an opinion one way or another here -- we can easily create an Iter8 task to write back this intermediate traffic split information, and it is up to the user if they want to use this task or not. If they choose to use this task, Iter8 experiment will use a FixedSplit deployment pattern (i.e., it will not actively alter cluster state even transiently). If they choose not to use this task, Iter8 experiment can use either a progressive deployment pattern or a fixed split deployment pattern (up to the end-user; Iter8 supports both).

[huang195]

@sriumcp Let me summarize these problems here, copied from our Slack discussion:

1. Problem 1 is simply stating that without iter8 writing back to the Env repo, it will not maintain GitOps guarantee, as what runs in the cluster could deviate from that of Env repo. Another facet of Problem 1 is how to not hard code things like env repo URL, directory path, etc. in the handler code.

2. Problem 2 says when Iter8 writes back to the Env repo, we are now introducing a race condition as now both CI and iter8 need to write to the same repo / files, which could lead to corruption. Slides 20-25 give a detailed example of how this type of corruption could happen

3. Problem 3 is regarding failure of handler code that change persistent state. If such code crashes in the middle, this could lead to inconsistency in persistent state. Who's responsible for recover from such failures?

4. Problem 4 is regarding whether or not iter8 should support both i) allow users to test every commit, and ii) allow users to test only the latest commit. My finding is it might support the former, but doesn't support the latter. Do we want to support both modes or just the former?

5. Problem 5 is regarding fine-grained sync. The use case you and Ang discussed about having iter8 sync'ing back intermediate state of an experiment has the advantage of faster recovery from a crashed experiment, but if each sync requires a separate PR, that could become a problem for human operators who need to manually merge each one.

[sriumcp]

A few additional comments...

0. The design above is intended for both GitOps and non-gitops CD pipelines. It is influenced by Option 1 presented by @huang195 (slides not included here). A subset of resources in the env repo can be modified by both CI and Iter8 (but there are invariants guaranteed to hold despite this fact). Please see above.
1. In the case of Istio deployments, it is likely that CI builds the container image (for Iter8 purposes it doesn't matter; image has been built and there's now a trigger kicking off experiment generation; this is the part of the CI pipeline which Iter8 is concerned with). For KFServing, the trigger is a model upload (probably to a COS bucket)... not an image build; the CI will construct an InferenceService for the candidate.

[sriumcp]

I gave a very detailed example on slides 20-25 that is showing how a race condition between CI and iter8 handler could cause corruption in the Env repo.

Yes, this is a valid concern, and I now fully understand slides 20-25. However, for the case of pre-emptive experiments described above, the above problem is indeed solved by the design. It is intended to guarantee the invariants described above (in the solution to problem 2). In particular, slide 24 is not how this design works...

[sriumcp]

Problem 3: A common deployment pattern is to put configurations of different environments in different Env repos, e.g., staging, prod. After an experiment is finished, the handler can be used to i) update the baseline in the staging Env repo, and ii) promote this version to the prod Env repo. As these are in 2 separate git repos, this would require 2 git commits. If crash happens between the 2 commits, you will have an inconsistent state.

Multi-cluster is certainly an interesting use-case. However, I don't believe multi-cluster GitOps is all that common. Flux and ArgoCD do have story around multi-cluster gitops, but from what I know, it is a very recent one.

Perhaps, it would be nice to start by solving the single-cluster (prod) gitops issue to begin with in a simple way, instead of tackling all the complexity all at once?

The design proposal above is intended to solve the single-cluster (prod) gitops use case.

[sriumcp]

I don't fully understand the response, but ...

Hopefully, it will be clearer during our webex.

[sriumcp]

Problem 5: I'm simply pointing out that the use of PRs for syncing intermediate states could result in many PRs getting created, and appears to be counter-productive for teams who want to leverage automation provided by CI/CD pipelines.

I agree. In the proposal described above, although VS is in the env-repo, it is not fully synced, only partially synced. Specifically, in ArgoCD for instance, you can essentially ask the gitops controller to ignore sync of specific fields within a resource... this is exactly how ArgoCD with deployments have auto-scaling with HPA enabled (replica count in deployment objects is ignored during sync and owned by HPA). Something similar for traffic splitting is useful during release automation as well (and I believe ArgoRollouts uses this mechanism today).

I do need to understand what is the FluxCD equivalent here (hopefully, there is one).

In terms of problem 5, it means Iter8 can claim ownership of transient traffic shifts during release, and only create the final promotion PR.

Again, this is merely the option used in the above proposal...

I don't believe there is "one gitops design" that fits all... I am particularly interested in the (new) handler task definitions that can enable different types of gitops solutions with Iter8. The (new) tasks described (imagined) in the experiment above are intended to solve the limited problem of pre-emptive experimentation with a single prod cluster.

[huang195]

Let's talk tomorrow. It's going to be more clear when we have a shared screen to point to.

[huang195]

@sriumcp Can you flesh out the details for step 3 in the gitops scenario? What does iter8 need to do to "release stable version" as the finish handler action?

[huang195]

@sriumcp @kalantar @fabolive
Here's a summary of our call to address the specific problems:

Problem 1: Regarding parameters related to env repo and directory path, it was decided that this is the best we can do at the moment, so no action needs to be taken.

Problem 2: Regarding race condition, not completely discussed before the meeting ended.

Problem 3: Regarding transaction-awareness of handler code, it was decided that it will be the responsibility of the handler to be atomic, e.g., two git commits in the same handler does not satisfy this requirement. Moreover as an action can have multiple tasks, all the tasks in the action need to act as an atomic operation. The action is to clearly document this constraint.

Problem 4: Regarding whether or not we should support both i) test every commit and ii) test only the newest one, the decision is to only support the latter. The former is deferred for now.

Problem 5: Regarding sync'ing intermediate states while an experiment is running, ran out of time and did not discuss.

[huang195]

Re: multiple tasks, in the simplest example, you could have 2 tasks, first one commits to the staging Env repo, and second one commits to the prod Env repo. Don't take this example too literally, as there could be other use cases. The point is if a crash happens after the first task finishes but before the second one finishes, we could be stuck in an inconsistent state that would require manual fixing to recover.

Re: gitops/apply, after looking at the handler repo, my feeling is we should rather stick with common/exec or common/script instead. It's just a couple lines of shell scripts, which is much easier to understand what it is doing rather going through layers of Golang wrappers. When someone wants to modify its behavior, it's also easier to modify the shell script rather than having to recompile Go code. Just my 2 cents.

+1 on clarify on nomenclature.

[sriumcp]

I feel the common/exec task doesn't have any atomicity guarantees inherently associated with it. If we do implement gitops/apply, I believe we can implement it in a way that guarantees atomicity (at least at the task level)... this can also be done as part of common/exec with a well written set of shell commands, but the burden is now on the creator of the experiment to think through this (instead of using the gitops/apply which comes with this guarantee).

We can take this a step further and say, an action can have a single gitops/apply task, not two; further, it needs to be preceded by a gitops/init task for the action to be valid and so on (a check which can be enforced by Iter8; at the very least, Iter8 can figure out that the action is not written properly, and can choose to terminate the experiment with a failure and notify with a reason).

In essence, by wrapping the functionality into well-defined atomic tasks, we can attach additional semantics to it. It is of course up to us to imagine what type of validation we would like to enforce.

---

TBH, as a starting point, I wouldn't hesitate to implement everything using common/exec (no atomicity or any other guarantee to begin with); this would get us the end-to-end GitOps prototype working to begin with (under the assumption that handler is written properly); once we do have the common/exec gitops task, we can certainly ask if there's a simpler way to encapsulate this in a task (under the covers, the Golang wrapper can actually use the common/exec task if that makes the implementation quicker)... anyway, these are not so much design but implementation details.

[sriumcp]

Re: multi-cluster / multi-repo GitOps release automation, I admit, I didn't give it much thought... minimum viability / lovability is what I was shooting for when thinking through the design (single env-repo; single cluster; pre-emptive, two-version experiments namely canary and A/B testing experiments).

[sriumcp]

Incidentally, in GitHub Actions, you can have multiple steps within a job, and multiple jobs within a workflow.

AFAIK, I don't believe it provides any type of atomicity guarantee at the step level (let alone the job / workflow level).

And developers seem comfortable using GH actions for CI/CD without these guarantees... which does make me wonder just how important the atomicity guarantees are to begin with (GH actions can fail too... as can GH). I do believe this is nice to have eventually, but is this a concern for the MVP?

Perhaps https://iter8.tools/0.5/reference/tasks/#error-handling-in-tasks -- this statement about errors in tasks might be sufficient to begin with?

[huang195]

Re: common/exec vs. gitops/*, maybe I don't fully understand what benefit the latter offers in terms of atomicity guarantee compared to the former. We can discuss this next week.

Re: Github Actions, that's a good point. The last time I checked Github Actions, I don't remember seeing any mentioning of atomicity or idempotent guarantee. If this is an acceptable practice, we can brush it aside for now.

My only remaining concern is problem 2 regarding race condition. When you get a chance to flesh up Step 3 of your design, I'll take a look and see if the race condition example I had in my deck is still applicable.

[sriumcp]

Pre-emptive GitOps experiment (take 2)

Productpage candidate deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-candidate # note the -candidate suffix. This is required.
  namespace: bookinfo-iter8
  labels:
    app: productpage
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v2
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: productpage # same as baseline
        version: v2 # your app version (can be v2, v3, v4, ... and so on as you upgrade)
        iter8AppVersion: candidate # always candidate (for the stable version, `candidate` is replaced with `stable`)
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: iter8/productpage:v2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        env:
          - name: deployment
            value: "productpage-v2"
          - name: namespace
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: color
            value: "green"
          - name: reward_min
            value: "10"
          - name: reward_max
            value: "20"
          - name: port
            value: "9080"

---

VS and DR

# This resource is updated by both CD pipeline segment and Iter8
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: bookinfo-iter8
  annotations:
    hash: 382ud18s # random hash to force git sync
spec:
  gateways:
  - mesh
  - bookinfo-gateway
  hosts:
  - productpage
  - "bookinfo.example.com"
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
        subset: stable
      weight: 100 # can go out of sync
    - destination:
        host: productpage
        port:
          number: 9080
        subset: candidate
      weight: 0 # can go out of sync
---
# This is resource is created once per application (not once per version)
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: bookinfo
  namespace: bookinfo-iter8
spec:
  host: productpage
  subsets:
  - labels:
      app: productpage
      iter8AppVersion: stable
    name: stable
  - labels:
      app: productpage
      iter8AppVersion: candidate
    name: candidate

---

apiVersion: iter8.tools/v2alpha2
kind: Experiment
metadata:
  name: quickstart-exp-7392lasdhf
  namespace: bookinfo-iter8
spec:
  # target identifies the service under experimentation using its fully qualified name
  target: bookinfo-iter8/productpage
  strategy:
    # this experiment will perform an A/B test
    testingPattern: A/B
    # this experiment will progressively shift traffic to the winning version
    deploymentPattern: Progressive
    actions:
      # when the experiment completes, promote the winning version using kubectl apply
      start:
      - task: gitops/init # drop this task for clusterops
        with:
          repo: github.com/bookinfo/productpage
          folder: env/repo/folder/path # optional; without this field, root of the repo is the default folder
          secret: ns/git-token
      - task: gitops/verify-readiness
        with:
          timeout: 1h
      - task: istio/verify-readiness
        with:
          timeout: 1h
          name: productpage
          namespace: bookinfo-iter8
      finish:
      - task: gitops/init
        with:
          repo: github.com/bookinfo/productpage
          folder: env/repo/folder/path # optional; without this field, root of the repo is the default folder
          secret: ns/git-token
      - task: gitops/update-istio-deployment
        with:
          name: productpage
          namespace: bookinfo-iter8
  criteria:
    rewards:
    # (business) reward metric to optimize in this experiment
    - metric: iter8-istio/user-engagement 
      preferredDirection: High
    objectives: # used for validating versions
    - metric: iter8-istio/mean-latency
      upperLimit: 300
    - metric: iter8-istio/error-rate
      upperLimit: "0.01"
    requestCount: iter8-istio/request-count
  duration: # product of fields determines length of the experiment
    intervalSeconds: 10
    iterationsPerLoop: 10
  versionInfo:
    # information about the app versions used in this experiment
    baseline:
      name: productpage
      weightObjRef:
        apiVersion: networking.istio.io/v1beta1
        kind: VirtualService
        namespace: bookinfo-iter8
        name: bookinfo
        fieldPath: .spec.http[0].route[0].weight
    candidates:
    - name: productpage-candidate
      weightObjRef:
        apiVersion: networking.istio.io/v1beta1
        kind: VirtualService
        namespace: bookinfo-iter8
        name: bookinfo
        fieldPath: .spec.http[0].route[1].weight

[sriumcp]

1. Cleanup of candidate deployment is possible, not included here....
2. gitops/update-istio-deployment updates the VS (100-0; random hash) and baseline deployment. It creates baseline deployment by dropping -candidate suffix, and setting iter8AppVersion: stable annotation... result is the a PR with the following changes...

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage # note the -candidate suffix is now dropped, since this is the stable version.
  namespace: bookinfo-iter8
  labels:
    app: productpage
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v2
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: productpage # same as baseline
        version: v2 # your app version (can be v2, v3, v4, ... and so on as you upgrade)
        iter8AppVersion: stable
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: iter8/productpage:v2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        env:
          - name: deployment
            value: "productpage-v2"
          - name: namespace
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: color
            value: "green"
          - name: reward_min
            value: "10"
          - name: reward_max
            value: "20"
          - name: port
            value: "9080"
---
# This resource is updated by both CD pipeline segment and Iter8
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: bookinfo-iter8
  annotations:
    hash: 8372nksdf # random hash to force git sync
spec:
  gateways:
  - mesh
  - bookinfo-gateway
  hosts:
  - productpage
  - "bookinfo.example.com"
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
        subset: stable
      weight: 100 # can go out of sync
    - destination:
        host: productpage
        port:
          number: 9080
        subset: candidate
      weight: 0 # can go out of sync

[sriumcp]

Claim 1: Git baseline = winner (PR merged) from the last completed experiment
Claim 2: Git candidate = candidate from latest CD pipeline segment run
Claim 3: Traffic split (in cluster) is always between 100 -- between baseline & candidate (in cluster)