E2E test framework #443

YaoZengzeng · 2024-06-18T02:01:07Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

E2E test framework to make the project more robust

Which issue(s) this PR fixes:
Fixes most part of #137

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

LiZhenCheng9527 · 2024-06-18T03:16:06Z

deploy/yaml/kmesh.yaml

@@ -54,7 +54,7 @@ spec:
                  path: istio-token
      containers:
        - name: kmesh
-          image: ghcr.io/kmesh-net/kmesh:latest


It's better to open a new folder for e2e's yaml

Good catch, I should find a way to configure HUB of the image

How do you install, if using helm we can customize the image

Now I apply yamls directly for convenience and I will use helm later.

LiZhenCheng9527 · 2024-06-18T03:26:42Z

test/e2e/run_test.sh

+# Exit immediately for non zero status
+set -e
+
+DEFAULT_KIND_IMAGE="kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e"


If you limit the kind/node image version, you will also have a limit on the kind version.
I have another option：Check for kind, then create cluster without specifying image.

Istio uses this to test on compatability with different k8s version.

What we need maybe add a mechanism to test comaptability with differen istio versions

LiZhenCheng9527 · 2024-06-18T03:29:33Z

test/e2e/run_test.sh

+    build_and_push_images
+fi
+
+go test -v -tags=integ ./test/e2e/...


Suggested change

go test -v -tags=integ ./test/e2e/...

go test -v -tags=integ $ROOT_DIR/test/e2e/...

ROOT_DIR=$(git rev-parse --show-toplevel)
Preventing test failures by not running in the kmesh root directory.

hzxuzhonghu

Looked just a small part

hzxuzhonghu · 2024-06-20T02:26:21Z

docs/proposal/e2e.md

+
+### Motivation
+
+E2E testing is a software approach that tests an application's workflow from start to finish, simulating read-user scenarios. The main purpose of E2E testing is to validate the system as a whole,ensuring that all the individual components and integrations work together seamlessly. It helps to identify any issues or defects that may arise from the interation between different components of the appliction, ensuing the application works as expected under normal operating conditions.


s/defects/detects

simulating read-user scenarios. what does this mean?

s/defects/detects

It is defetcs actually :)

simulating read-user scenarios. what does this mean?

updated, it is real user scenarios.

hzxuzhonghu · 2024-06-24T01:35:50Z

docs/proposal/e2e.md

+
+1. Use [namespace](https://github.com/istio/istio/blob/master/pkg/test/framework/components/namespace/namespace.go) package to create namespace for deploying test applications.
+
+2. Use [deployment](https://github.com/istio/istio/blob/master/pkg/test/framework/components/echo/deployment/builder.go) package to build test applications. `WithClusters()` can be used to specify the clusters where the test applications should be deployed. Each call to `WithConfig()` will generate a test application with corresponding configuration. Actually we create all test applications at the beginning. In each test case, we select some suitable applications for testing. And an application can be used as both a client and a server.


Should document if these apps do not satisfy new tests, how can developer deploy new apps

hzxuzhonghu · 2024-06-24T01:39:01Z

go.mod

@@ -5,6 +5,8 @@ go 1.22.0
 // Client-go does not handle different versions of mergo due to some breaking changes - use the matching version
 replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.5

+replace istio.io/api => istio.io/api v1.22.0-alpha.1.0.20240620154034-5b788fec62d2


why replace instead of bumping directly

If we bump directly, it will use version istio.io/api v1.22.1, there will be some interface incompatible

I donot understand what do you mean

hzxuzhonghu · 2024-06-24T01:40:53Z

test/e2e/baseline_test.go

@@ -0,0 +1,155 @@
+//go:build integ


do we really need this building tag

It will be added automaticlly in my vscode

hzxuzhonghu · 2024-06-24T01:47:11Z

test/e2e/run_test.sh

+# Exit immediately for non zero status
+set -e
+
+DEFAULT_KIND_IMAGE="kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e"


Istio uses this to test on compatability with different k8s version.

What we need maybe add a mechanism to test comaptability with differen istio versions

hzxuzhonghu · 2024-06-24T01:52:41Z

test/e2e/run_test.sh

+    fi
+
+    # Create KinD cluster.
+    cat <<EOF | kind create cluster --name="${NAME}" -v4 --retain --image "${IMAGE}" --config=-


can we add a multi nodes cluster, since we have daemonset

hzxuzhonghu · 2024-06-24T01:59:29Z

test/e2e/main_test.go

+
+	builder := deployment.New(t).
+		WithClusters(t.Clusters()...).
+		/*


hzxuzhonghu · 2024-06-24T02:01:05Z

test/e2e/main_test.go

+	// AllWaypoint is a waypoint for all types
+	AllWaypoint echo.Instances
+	// WorkloadAddressedWaypoint is a workload only waypoint
+	WorkloadAddressedWaypoint echo.Instances
+	// ServiceAddressedWaypoint is a service only waypoint
+	ServiceAddressedWaypoint echo.Instances


I am not sure this is suitable for kmesh, at least the names and comments are confusing

removed, we don't need these fields now.

hzxuzhonghu · 2024-06-24T02:01:50Z

test/e2e/main_test.go

+	Captured                  = "captured"
+	Uncaptured                = "uncaptured"
+	WaypointImageAnnotation   = "sidecar.istio.io/proxyImage"
+	KmeshCustomWaypointImage  = "ghcr.io/kmesh-net/waypoint-x86:v0.3.0"


should support customize it

hzxuzhonghu · 2024-06-24T02:05:47Z

test/e2e/baseline_test.go

+		}
+		if src.Config().IsUncaptured() {
+			// TODO: fix this and remove this skip
+			t.Skip("https://github.com/istio/istio/issues/43238")


The test case is direclty copied from ambient integration test (https://github.com/istio/istio/blob/master/tests/integration/ambient/baseline_test.go#L628`).

Actually I think most the test cases there should be copied and used direcly. Because we need to fully match ambient's capabilities and be transparent to isito and users.

hzxuzhonghu · 2024-06-24T02:06:09Z

test/e2e/baseline_test.go

+		if opt.Scheme != scheme.HTTP {
+			return
+		}
+		if !dst.Config().HasServiceAddressedWaypointProxy() {


Can you remove this istio special concept

Signed-off-by: YaoZengzeng <[email protected]>

hzxuzhonghu

looks super good, i will try with it. And understand it deeper

test/e2e/run_test.sh

hzxuzhonghu · 2024-06-27T02:16:05Z

test/e2e/run_test.sh

+fi
+
+if [[ -z "${SKIP_SETUP:-}" ]]; then
+    setup_kind_cluster


If i have already installed kind cluster and istio, how can i reuse them

You can use flags to skip some steps, the specific usage method has been added to the proposal.

Signed-off-by: YaoZengzeng <[email protected]>

YaoZengzeng · 2024-06-27T09:17:53Z

@hzxuzhonghu PTAL

hzxuzhonghu

/lgtm

kmesh-bot · 2024-06-27T09:26:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kmesh-bot added do-not-merge/work-in-progress kind/feature labels Jun 18, 2024

kmesh-bot requested review from hzxuzhonghu and supercharge-xsy June 18, 2024 02:01

kmesh-bot added the size/XXL label Jun 18, 2024

LiZhenCheng9527 reviewed Jun 18, 2024

View reviewed changes

YaoZengzeng force-pushed the e2e branch 4 times, most recently from 1438e87 to 52777b8 Compare June 18, 2024 07:50

YaoZengzeng mentioned this pull request Jun 18, 2024

Bump istio #445

Merged

YaoZengzeng force-pushed the e2e branch 2 times, most recently from 66ff584 to 59a61a6 Compare June 19, 2024 01:52

kmesh-bot added size/XL and removed size/XXL labels Jun 19, 2024

YaoZengzeng force-pushed the e2e branch from 59a61a6 to 54f7afe Compare June 19, 2024 02:12

kmesh-bot added size/XXL and removed size/XL labels Jun 19, 2024

YaoZengzeng force-pushed the e2e branch 4 times, most recently from 1fb6e0a to 8a7ffc4 Compare June 21, 2024 09:22

hzxuzhonghu previously approved these changes Jun 22, 2024

View reviewed changes

kmesh-bot added the approved label Jun 22, 2024

hzxuzhonghu removed the approved label Jun 22, 2024

YaoZengzeng dismissed hzxuzhonghu’s stale review via 6cf790e June 22, 2024 03:17

YaoZengzeng force-pushed the e2e branch from 6cf790e to c8d2502 Compare June 22, 2024 03:19

YaoZengzeng changed the title ~~[WIP] E2E test framework~~ E2E test framework Jun 22, 2024

kmesh-bot removed the do-not-merge/work-in-progress label Jun 22, 2024

hzxuzhonghu reviewed Jun 24, 2024

View reviewed changes

YaoZengzeng added 15 commits June 26, 2024 08:55

e2e support kmesh install

63f044f

Signed-off-by: YaoZengzeng <[email protected]>

add app

22d0ce4

Signed-off-by: YaoZengzeng <[email protected]>

deploy test case

c7a8ee2

Signed-off-by: YaoZengzeng <[email protected]>

enable deploying waypoint in kmesh

0610c2c

Signed-off-by: YaoZengzeng <[email protected]>

pass first e2e test

541612a

Signed-off-by: YaoZengzeng <[email protected]>

setup part of local registry

57720cf

Signed-off-by: YaoZengzeng <[email protected]>

only pull kmesh image from local registry

968c923

Signed-off-by: YaoZengzeng <[email protected]>

add github action workflow

33c6663

Signed-off-by: YaoZengzeng <[email protected]>

part of e2e proposal

dade647

Signed-off-by: YaoZengzeng <[email protected]>

part II of e2e proposal

3dcc628

Signed-off-by: YaoZengzeng <[email protected]>

use helm to install Kmesh

d69fb73

Signed-off-by: YaoZengzeng <[email protected]>

fix comments

18062ae

Signed-off-by: YaoZengzeng <[email protected]>

fix comments

508dfa4

Signed-off-by: YaoZengzeng <[email protected]>

fix comments

5d9141f

Signed-off-by: YaoZengzeng <[email protected]>

install istio and kmesh in script

ab1ca85

Signed-off-by: YaoZengzeng <[email protected]>

YaoZengzeng force-pushed the e2e branch from ff3ffb1 to ab1ca85 Compare June 26, 2024 08:55

hzxuzhonghu reviewed Jun 26, 2024

View reviewed changes

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

test/e2e/run_test.sh Outdated Show resolved Hide resolved

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

test/e2e/run_test.sh Outdated Show resolved Hide resolved

hzxuzhonghu reviewed Jun 27, 2024

View reviewed changes

YaoZengzeng added 2 commits June 27, 2024 07:00

fix comments

f3f75c0

Signed-off-by: YaoZengzeng <[email protected]>

config local registry when setup cluster

f011a72

Signed-off-by: YaoZengzeng <[email protected]>

YaoZengzeng force-pushed the e2e branch from f7ea7bc to f011a72 Compare June 27, 2024 08:11

hzxuzhonghu approved these changes Jun 27, 2024

View reviewed changes

kmesh-bot assigned hzxuzhonghu Jun 27, 2024

kmesh-bot added the lgtm label Jun 27, 2024

kmesh-bot added the approved label Jun 27, 2024

kmesh-bot merged commit ba03077 into kmesh-net:main Jun 27, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E test framework #443

E2E test framework #443

YaoZengzeng commented Jun 18, 2024

LiZhenCheng9527 Jun 18, 2024

YaoZengzeng Jun 18, 2024

hzxuzhonghu Jun 20, 2024

YaoZengzeng Jun 20, 2024

LiZhenCheng9527 Jun 18, 2024

hzxuzhonghu Jun 24, 2024

LiZhenCheng9527 Jun 18, 2024 •

edited

Loading

hzxuzhonghu left a comment

hzxuzhonghu Jun 20, 2024

hzxuzhonghu Jun 20, 2024

YaoZengzeng Jun 22, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu Jun 24, 2024

YaoZengzeng Jun 24, 2024

hzxuzhonghu Jun 26, 2024

hzxuzhonghu Jun 24, 2024

YaoZengzeng Jun 24, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu Jun 24, 2024

YaoZengzeng Jun 24, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu Jun 24, 2024

YaoZengzeng Jun 24, 2024

hzxuzhonghu Jun 24, 2024

hzxuzhonghu left a comment

hzxuzhonghu Jun 27, 2024

YaoZengzeng Jun 27, 2024

YaoZengzeng commented Jun 27, 2024

hzxuzhonghu left a comment

kmesh-bot commented Jun 27, 2024

	go test -v -tags=integ ./test/e2e/...
	go test -v -tags=integ $ROOT_DIR/test/e2e/...


		### Motivation

		E2E testing is a software approach that tests an application's workflow from start to finish, simulating read-user scenarios. The main purpose of E2E testing is to validate the system as a whole,ensuring that all the individual components and integrations work together seamlessly. It helps to identify any issues or defects that may arise from the interation between different components of the appliction, ensuing the application works as expected under normal operating conditions.


		1. Use [namespace](https://github.com/istio/istio/blob/master/pkg/test/framework/components/namespace/namespace.go) package to create namespace for deploying test applications.

		2. Use [deployment](https://github.com/istio/istio/blob/master/pkg/test/framework/components/echo/deployment/builder.go) package to build test applications. `WithClusters()` can be used to specify the clusters where the test applications should be deployed. Each call to `WithConfig()` will generate a test application with corresponding configuration. Actually we create all test applications at the beginning. In each test case, we select some suitable applications for testing. And an application can be used as both a client and a server.

E2E test framework #443

E2E test framework #443

Conversation

YaoZengzeng commented Jun 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LiZhenCheng9527 Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YaoZengzeng commented Jun 27, 2024

hzxuzhonghu left a comment

Choose a reason for hiding this comment

kmesh-bot commented Jun 27, 2024

LiZhenCheng9527 Jun 18, 2024 •

edited

Loading